Return to Answer

That p-value doesn't really take the structure of the confusion matrix into account, however. It doesn't recognize that the correct classifications are of two different types (and similarly for the incorrect classifications). Furthermore, it doesn't recognize that the underlying data are non-independent, and therefore doesn't deal with the non-independence appropriately (from a statistical point of view). The test that does recognize these things is McNemar's test. I discuss McNemar's test here and here, which you may want to read to get a fuller overview. Your last highlighted line is the p-value from McNemar's test of the confusion matrix.

That p-value only tells you that your accuracy is better, you may also want to know if your model is biased. That is, does your classifier tend to say B / 1 more often than is really the case? You could test the proportion the model says B against the observed proportion B, but that wouldn't really take the structure of the confusion matrix into account, however. It wouldn't recognize that the underlying data are non-independent, and therefore doesn't deal with the non-independence appropriately (from a statistical point of view). The test that does recognize these things is McNemar's test. I discuss McNemar's test here and here, which you may want to read to get a fuller overview. Your last highlighted line is the p-value from McNemar's test of the confusion matrix.

That p-value doesn't really take the structure of the confusion matrix into account, however. It doesn't recognize that the correct classifications are of two different types (and similarly for the incorrect classifications). Furthermore, it doesn't recognize that the underlying data are non-independent, and therefore doesn't deal with the non-independence appropriately (from a statistical point of view). The test that does recognize these things is McNemar's test. I discuss McNemar's test here and here, which you may want to read to get a fuller overview. Your last highlighted line is the p-value from McNemar's test of the confusion matrix.

That p-value doesn't really take the structure of the confusion matrix into account, however. It doesn't recognize that the correct classifications are of two different types (and similarly for the incorrect classifications). Furthermore, it doesn't recognize that the underlying data are non-independent, and therefore doesn't deal with the non-independence appropriately (from a statistical point of view). The test that does recognize these things is McNemar's test. I discuss McNemar's test here and here, which you may want to read to get a fuller overview. Your last highlighted line is the p-value from McNemar's test of the confusion matrix.

library(caret)

mat = matrix(c(55,34,56,255), ncol=2, byrow=TRUE)
mat
#      [,1] [,2]
# [1,]   55   34
# [2,]   56  255
confusionMatrix(as.table(mat), positive="B")
# Confusion Matrix and Statistics
# 
#     A   B
# A  55  34
# B  56 255
#                                          
#                Accuracy : 0.775          
#                  95% CI : (0.7309, 0.815)
#     No Information Rate : 0.7225         
#     P-Value [Acc > NIR] : 0.00994        
#                                          
#                   Kappa : 0.4024         
#  Mcnemar's Test P-Value : 0.02686        
#                                          
#             Sensitivity : 0.8824         
#             Specificity : 0.4955         
#          Pos Pred Value : 0.8199         
#          Neg Pred Value : 0.6180         
#              Prevalence : 0.7225         
#          Detection Rate : 0.6375         
#    Detection Prevalence : 0.7775         
#       Balanced Accuracy : 0.6889         
#                                          
#        'Positive' Class : B              

This comes from the caret package in R. It presents a confusion matrix, which is a contingency table of the predicted and actual classes from some classifier, with some information about the confusion matrix that can help you interpret different aspects of the quality of the classifier.

library(caret)

mat = matrix(c(55,34,56,255), ncol=2, byrow=TRUE)
mat  # output omitted
confusionMatrix(as.table(mat), positive="B")
# Confusion Matrix and Statistics
# 
#     A   B
# A  55  34
# B  56 255
#                                          
#                Accuracy : 0.775          
#                  95% CI : (0.7309, 0.815)
#     No Information Rate : 0.7225         
#     P-Value [Acc > NIR] : 0.00994        
#                                          
#                   Kappa : 0.4024         
#  Mcnemar's Test P-Value : 0.02686        
#                                          
#             Sensitivity : 0.8824         
#             Specificity : 0.4955         
#          Pos Pred Value : 0.8199         
#          Neg Pred Value : 0.6180         
#              Prevalence : 0.7225         
#          Detection Rate : 0.6375         
#    Detection Prevalence : 0.7775         
#       Balanced Accuracy : 0.6889         
#                                          
#        'Positive' Class : B              

This comes from the caret package in R. It presents a confusion matrix, which is a contingency table of the predicted and actual classes from some classifier, with some information about the confusion matrix that can help you interpret different aspects of the quality of the classifier.

The Accuracy is the proportion of the cases that were classified correctly. You can see that there were 400 cases, and the classifier correctly said A and B (in your case 0 and 1) 55 and 255 times, respectively. Thus, there were 310 correct classifications out of 400 for 77.5% correct. Because each classification is either correct or incorrect, that value is a binomial and we can form a confidence interval for it, just as we can for any other binomial. There are lots of ways to calculate confidence intervals for a binomial.

Because 289 of the 400 cases were in the 'positive' class, you could just say "B" for every cases without fitting any classifier and still get 72.25% correct. That doesn't sound so bad, but in fact, it means you don't know anything relevant to the situation at all, so we want to take that into account. That is called the "no information rate". Before you think your model is accurate, or provides valuable information, you want your accuracy to be greater than that. You might further want to know if your accuracy doesn't just happen to be above that level, but if it is significantly greater than the no information rate. A test of whether an observed proportion is greater than some specified value is a simple one-tailed binomial test. The p-value from which is what the P-Value [Acc > NIR] displays.

That p-value doesn't really take the structure of the confusion matrix into account, however. It doesn't recognize that the correct classifications are of two different types (and similarly for the incorrect classifications). Furthermore, it doesn't recognize that the underlying data are non-independent, and therefore doesn't deal with the non-independence appropriately (from a statistical point of view). The test that does recognize these things is McNemar's test. I discuss McNemar's test here and here, which you may want to read to get a fuller overview. Your last highlighted line is the p-value from McNemar's test of the confusion matrix.