Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 2 down vote favorite
1

I want to compare the accuracy of two classifiers for statistical significance. Both classifiers are run on the same data set. This leads me to believe I should be using a one sample t-test from what I have been reading.

For example:

Classifier 1: 51% accuracy
Classifier 2: 64% accuracy
Dataset size: 78,000

Is this the right test to be using? If so how do I calculate if the difference in accuracy between classifier is significant?

Or should I be using another test?

share|improve this question

2 Answers

up vote 3 down vote

I would probably opt for McNemar's test if you only train the classifiers once. David Barber also suggests a rather neat Bayesian test that seems rather elegant to me, but isn't widely used (it is also mentioned in his book).

Just to add, as Peter Flom says, the answer is almost certainly "yes" just by looking at the difference in performance and the size of the sample (I take the figures quoted are test set performance rather than training set performance).

Incidentally Japkowicz and Shah have a recent book out on "Evaluating Learning Algorithms: A Classification Perspective", I haven't read it, but it looks like a useful reference for these sorts of issues.

share|improve this answer
2  
The accuracies for each fold will not be independent, which will violate the assumptions of most statistical tests, but probably won't be a big issue. I often use 100 random training/test splits and then use the Wilcoxon paired signed rank test (use the same random splits for both classifiers). I prefer that sort of test as I often use small datasets (as I am interested in overfitting) so the variability between random splits tends to be comparable to the difference in performance between classifiers. – Dikran Marsupial Apr 11 '12 at 15:07
1  
(+1) for Wilcoxon paired signed rank test (and the link to the book ... if the toc can fulfill its promises this book can become a must-read of all MLs :O) – steffen Apr 12 '12 at 6:57
1  
@steffen yes I have been meaning to get a copy of the book for a while; proper performance evaluation is not as common as it should in machine learning work, and has been a component of my research interests for a while. – Dikran Marsupial Apr 12 '12 at 13:03
1  
BTW, my copy of the book has arrived, all I need to do now is find the time to read it! – Dikran Marsupial May 1 '12 at 10:52
1  
I have also used signed rank tests as well as paired t-tests for comparing classifiers. However each time I report using a one-sided test for this purpose I get a hard time from reviewers so have reverted to using two-sided tests! – BGreene Jul 25 '12 at 15:15
show 1 more comment
up vote 1 down vote

I can tell you, without even running anything, that the difference will be highly statistically significant. It passes the IOTT (interocular trauma test - it hits you between the eyes).

If you do want to do a test, though, you could do it as a test of two proportions - this can be done with a two sample t-test.

You might want to break "accuracy" down into its components, though; sensitivity and specificity, or false-positive and false-negative. In many applications, the cost of the different errors are quite different.

share|improve this answer
Agreed - this will clearly be significant. Nitpick: You would use a $z$-test to test two proportions (approximately) - this has to do with the convergence of a binomial distribution to the normal as $n$ increases. See section 5.2 en.wikipedia.org/wiki/Statistical_hypothesis_testing – Macro Apr 11 '12 at 14:27
On second thought, a $t$-test may still be asymptotically valid, by the CLT, but there must a reason the $z$-test is usually used here. – Macro Apr 11 '12 at 14:28
The accuracy percentage I have put in my question are just an example. – Chris Apr 11 '12 at 14:45

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.