Is it possible to perform a statistical test to determine if one classifier is better than the other using only the confusion matrices of these classifiers?

What about the average accuracies from k-fold cross validation?

I have a number confusion matrices and average accuracy for classifiers obtained through k-fold cross validation (done using RapidMiner). The data sets for these classifiers are all the same, though the splitting into folds was all done independently. What I'd like to do is, given two of these classifiers, A and B, be able to test if A is statistically better than B using only the confusion matrices and/or the average accuracies for classifiers A and B.

All the statistical tests I've found so far require knowing the number of samples that A classified correctly when B did not, and vice-versa. (McNemar's Test for example). I can generate this data if necessary, but I'd like to avoid it if reasonably possible.


1 Answer 1


You want to test whether $$p_A-p_B>0,$$ where $$p_A, p_B$$ are the accuracies of the classifiers. To test this, you need an estimate of $$p_A-p_B$$ and $$Var(p_A-p_B)=Var(p_A)+Var(p_B)-2Cov(p_A,p_B)$$. Without knowing the samples that each classifier gets right/wrong, you won't be able to estimate the covariance, thus you can't statistically compare the classifiers.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.