I want to compare the accuracy of two classifiers for statistical significance. Both classifiers are run on the same data set. This leads me to believe I should be using a one sample t-test from what I have been reading.

For example:

Classifier 1: 51% accuracy
Classifier 2: 64% accuracy
Dataset size: 78,000

Is this the right test to be using? If so how do I calculate if the difference in accuracy between classifier is significant?

Or should I be using another test?

  • $\begingroup$ Hi @Chris, have you found a test to compare two models for statistical significance ? $\endgroup$
    – Gauranga
    2 days ago

I would probably opt for McNemar's test if you only train the classifiers once. David Barber also suggests a rather neat Bayesian test that seems rather elegant to me, but isn't widely used (it is also mentioned in his book).

Just to add, as Peter Flom says, the answer is almost certainly "yes" just by looking at the difference in performance and the size of the sample (I take the figures quoted are test set performance rather than training set performance).

Incidentally Japkowicz and Shah have a recent book out on "Evaluating Learning Algorithms: A Classification Perspective", I haven't read it, but it looks like a useful reference for these sorts of issues.

  • 1
    $\begingroup$ I am running 10-fold cross validation to get these results. Does that mean they are actually different data sets. That is the total size, which is split for test/train in cross validation $\endgroup$
    – Chris
    Apr 11 '12 at 14:43
  • 5
    $\begingroup$ The accuracies for each fold will not be independent, which will violate the assumptions of most statistical tests, but probably won't be a big issue. I often use 100 random training/test splits and then use the Wilcoxon paired signed rank test (use the same random splits for both classifiers). I prefer that sort of test as I often use small datasets (as I am interested in overfitting) so the variability between random splits tends to be comparable to the difference in performance between classifiers. $\endgroup$ Apr 11 '12 at 15:07
  • 2
    $\begingroup$ (+1) for Wilcoxon paired signed rank test (and the link to the book ... if the toc can fulfill its promises this book can become a must-read of all MLs :O) $\endgroup$
    – mlwida
    Apr 12 '12 at 6:57
  • 3
    $\begingroup$ I have also used signed rank tests as well as paired t-tests for comparing classifiers. However each time I report using a one-sided test for this purpose I get a hard time from reviewers so have reverted to using two-sided tests! $\endgroup$
    – BGreene
    Jul 25 '12 at 15:15
  • 3
    $\begingroup$ Given that OP clarified in the comments that the question was actually about cross-validation, would you perhaps consider expanding your answer to cover that topic? We can edit the Q then. This is an important topic and there are a couple of very related (or even duplicate) questions but none has a good answer. In a comment above you recommend using a paired test on the CV estimates and say that you don't think that non-independence is a big issue here. Why not? It sounds to me like a potentially massive issue! $\endgroup$
    – amoeba
    Nov 5 '15 at 23:45

I can tell you, without even running anything, that the difference will be highly statistically significant. It passes the IOTT (interocular trauma test - it hits you between the eyes).

If you do want to do a test, though, you could do it as a test of two proportions - this can be done with a two sample t-test.

You might want to break "accuracy" down into its components, though; sensitivity and specificity, or false-positive and false-negative. In many applications, the cost of the different errors are quite different.

  • $\begingroup$ Agreed - this will clearly be significant. Nitpick: You would use a $z$-test to test two proportions (approximately) - this has to do with the convergence of a binomial distribution to the normal as $n$ increases. See section 5.2 en.wikipedia.org/wiki/Statistical_hypothesis_testing $\endgroup$
    – Macro
    Apr 11 '12 at 14:27
  • $\begingroup$ On second thought, a $t$-test may still be asymptotically valid, by the CLT, but there must a reason the $z$-test is usually used here. $\endgroup$
    – Macro
    Apr 11 '12 at 14:28
  • 2
    $\begingroup$ The accuracy percentage I have put in my question are just an example. $\endgroup$
    – Chris
    Apr 11 '12 at 14:45
  • $\begingroup$ As I understand the question is not that much about the particular numbers provided in the example, but about a universal method allowing to statistically tell whether the performance difference in the two classifiers is significant or not. $\endgroup$
    – Data Man
    Aug 9 '20 at 11:22

Since accuracy, in this case, is the proportion of samples correctly classified, we can apply the test of hypothesis concerning a system of two proportions.

Let $\hat p_1$ and $\hat p_2$ be the accuracies obtained from classifiers 1 and 2 respectively, and $n$ be the number of samples. The number of samples correctly classified in classifiers 1 and 2 are $x_1$ and $x_2$ respectively.

$ \hat p_1 = x_1/n,\quad \hat p_2 = x_2/n$

The test statistic is given by

$\displaystyle Z = \frac{\hat p_1 - \hat p_2}{\sqrt{2\hat p(1 -\hat p)/n}}\qquad$ where $\quad\hat p= (x_1+x_2)/2n$

Our intention is to prove that the global accuracy of classifier 2, i.e., $p_2$, is better than that of classifier 1, which is $p_1$. This frames our hypothesis as

  • $H_0: p_1 = p_2\quad$ (null hypothesis stating both are equal)
  • $H_a: p_1 < p_2\quad$ (alternative hypotyesis claiming the newer one is better than the existing)

The rejection region is given by

$Z < -z_\alpha \quad$ (if true reject $H_0$ and accept $H_a$)

where $z_\alpha$ is obtained from a standard normal distribition that pertains to a level of significance, $\alpha$. For instance $z_{0.5} = 1.645$ for 5% level of significance. This means that if the relation $Z < -1.645$ is true, then we could say with 95% confidence level ($1-\alpha$) that classifier 2 is more accurate than classifier 1.


  1. R. Johnson and J. Freund, Miller and Freund’s Probability and Statistics for Engineers, 8th Ed. Prentice Hall International, 2011. (Primary source)
  2. Test of Hypothesis-Concise Formula Summary. (Adopted from [1])
  • $\begingroup$ Shouldn't $\quad\hat p$ be the average of $\hat p_1$ and $\hat p_2$? So the denominator should be 2n in $\quad\hat p= (x_1+x_2)/2n$. $\endgroup$
    – Shiva Tp
    Sep 30 '18 at 10:38
  • $\begingroup$ Though I agree that a test for proportions could be used, there is nothing in the original question that suggests a one-sided test is appropriate. Moreover, "we could say with 95% confidence" is a common misinterpretation. See e.g. here: metheval.uni-jena.de/lehre/0405-ws/evaluationuebung/haller.pdf $\endgroup$ Sep 30 '18 at 11:02
  • $\begingroup$ @ShivaTp Indeed. Thanks for pointing the much needed typo correction. Edit confirmed. $\endgroup$
    – Ébe Isaac
    Sep 30 '18 at 11:44
  • $\begingroup$ I like your answer. I wonder about the following though. With binary classification, in practice it is often the case that classes are inbalanced (positives being the minority) and so True Positives are of greater value than True Negatives. This may lead to conclusion that correctly classifying the minority class is a stronger signal of the classifier performance. I wonder whether in such scenario it would be still correct to use the test of proportions that you proposed? Because this test makes no distinction between TP and TN. It treats all those cases equally. $\endgroup$
    – Data Man
    Aug 9 '20 at 11:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.