# Comparing two classifier accuracy results for statistical significance with t-test

I want to compare the accuracy of two classifiers for statistical significance. Both classifiers are run on the same data set. This leads me to believe I should be using a one sample t-test from what I have been reading.

For example:

Classifier 1: 51% accuracy
Classifier 2: 64% accuracy
Dataset size: 78,000

Is this the right test to be using? If so how do I calculate if the difference in accuracy between classifier is significant?

Or should I be using another test?

-

I would probably opt for McNemar's test if you only train the classifiers once. David Barber also suggests a rather neat Bayesian test that seems rather elegant to me, but isn't widely used (it is also mentioned in his book).

Just to add, as Peter Flom says, the answer is almost certainly "yes" just by looking at the difference in performance and the size of the sample (I take the figures quoted are test set performance rather than training set performance).

Incidentally Japkowicz and Shah have a recent book out on "Evaluating Learning Algorithms: A Classification Perspective", I haven't read it, but it looks like a useful reference for these sorts of issues.

-
The accuracies for each fold will not be independent, which will violate the assumptions of most statistical tests, but probably won't be a big issue. I often use 100 random training/test splits and then use the Wilcoxon paired signed rank test (use the same random splits for both classifiers). I prefer that sort of test as I often use small datasets (as I am interested in overfitting) so the variability between random splits tends to be comparable to the difference in performance between classifiers. –  Dikran Marsupial Apr 11 '12 at 15:07
(+1) for Wilcoxon paired signed rank test (and the link to the book ... if the toc can fulfill its promises this book can become a must-read of all MLs :O) –  steffen Apr 12 '12 at 6:57
@steffen yes I have been meaning to get a copy of the book for a while; proper performance evaluation is not as common as it should in machine learning work, and has been a component of my research interests for a while. –  Dikran Marsupial Apr 12 '12 at 13:03
BTW, my copy of the book has arrived, all I need to do now is find the time to read it! –  Dikran Marsupial May 1 '12 at 10:52
I have also used signed rank tests as well as paired t-tests for comparing classifiers. However each time I report using a one-sided test for this purpose I get a hard time from reviewers so have reverted to using two-sided tests! –  BGreene Jul 25 '12 at 15:15

I can tell you, without even running anything, that the difference will be highly statistically significant. It passes the IOTT (interocular trauma test - it hits you between the eyes).

If you do want to do a test, though, you could do it as a test of two proportions - this can be done with a two sample t-test.

You might want to break "accuracy" down into its components, though; sensitivity and specificity, or false-positive and false-negative. In many applications, the cost of the different errors are quite different.

-
Agreed - this will clearly be significant. Nitpick: You would use a $z$-test to test two proportions (approximately) - this has to do with the convergence of a binomial distribution to the normal as $n$ increases. See section 5.2 en.wikipedia.org/wiki/Statistical_hypothesis_testing –  Macro Apr 11 '12 at 14:27
On second thought, a $t$-test may still be asymptotically valid, by the CLT, but there must a reason the $z$-test is usually used here. –  Macro Apr 11 '12 at 14:28
The accuracy percentage I have put in my question are just an example. –  Chris Apr 11 '12 at 14:45

Total accuracy is not a normed index. ESS (effect strength for sensitivity) is a normed index of classification accuracy on which 0 = classification accuracy expected by chance, and 100 = perfect, errorless classification. Here is a link to a free article that discusses this index: http://optimalprediction.com/files/pdf/V1A2.pdf

-