2
$\begingroup$

I want know the formulas for the information highlighted in the confusion matrix in the picture below:

enter image description here

$\endgroup$
  • $\begingroup$ Although the code & output depicted are from R, this isn't a programming question. The question asks for the formulas used to produce the highlighted information. Those formulas are mathematical &/or statistical in nature. This should be considered on topic here, IMO. $\endgroup$ – gung Dec 27 '16 at 21:22
  • $\begingroup$ Related: Definition of p-value in carets confusion matrix method. $\endgroup$ – gung Dec 27 '16 at 22:04
4
$\begingroup$
library(caret)

mat = matrix(c(55,34,56,255), ncol=2, byrow=TRUE)
mat  # output omitted
confusionMatrix(as.table(mat), positive="B")
# Confusion Matrix and Statistics
# 
#     A   B
# A  55  34
# B  56 255
#                                          
#                Accuracy : 0.775          
#                  95% CI : (0.7309, 0.815)
#     No Information Rate : 0.7225         
#     P-Value [Acc > NIR] : 0.00994        
#                                          
#                   Kappa : 0.4024         
#  Mcnemar's Test P-Value : 0.02686        
#                                          
#             Sensitivity : 0.8824         
#             Specificity : 0.4955         
#          Pos Pred Value : 0.8199         
#          Neg Pred Value : 0.6180         
#              Prevalence : 0.7225         
#          Detection Rate : 0.6375         
#    Detection Prevalence : 0.7775         
#       Balanced Accuracy : 0.6889         
#                                          
#        'Positive' Class : B              

This comes from the caret package in R. It presents a confusion matrix, which is a contingency table of the predicted and actual classes from some classifier, with some information about the confusion matrix that can help you interpret different aspects of the quality of the classifier.

The Accuracy is the proportion of the cases that were classified correctly. You can see that there were 400 cases, and the classifier correctly said A and B (in your case 0 and 1) 55 and 255 times, respectively. Thus, there were 310 correct classifications out of 400 for 77.5% correct. Because each classification is either correct or incorrect, that value is a binomial and we can form a confidence interval for it, just as we can for any other binomial. There are lots of ways to calculate confidence intervals for a binomial.

Because 289 of the 400 cases were in the 'positive' class, you could just say "B" for every cases without fitting any classifier and still get 72.25% correct. That doesn't sound so bad, but in fact, it means you don't know anything relevant to the situation at all, so we want to take that into account. That is called the "no information rate". Before you think your model is accurate, or provides valuable information, you want your accuracy to be greater than that. You might further want to know if your accuracy doesn't just happen to be above that level, but if it is significantly greater than the no information rate. A test of whether an observed proportion is greater than some specified value is a simple one-tailed binomial test. The p-value from which is what the P-Value [Acc > NIR] displays.

That p-value only tells you that your accuracy is better, you may also want to know if your model is biased. That is, does your classifier tend to say B / 1 more often than is really the case? You could test the proportion the model says B against the observed proportion B, but that wouldn't really take the structure of the confusion matrix into account, however. It wouldn't recognize that the underlying data are non-independent, and therefore doesn't deal with the non-independence appropriately (from a statistical point of view). The test that does recognize these things is McNemar's test. I discuss McNemar's test here and here, which you may want to read to get a fuller overview. Your last highlighted line is the p-value from McNemar's test of the confusion matrix.

$\endgroup$
  • $\begingroup$ +1 nice to see you answer these types of questions. $\endgroup$ – Haitao Du Aug 21 '17 at 16:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.