Based on estimated classification accuracy, I want to test whether one classifier is statistically better on a base set than another classifier . For each classifier, I select a training and testing sample randomly from the base set, train the model, and test the model. I do this ten times for each classifier. I therefore have ten estimate classification accuracy measurements for each classifier. How do I statistically test whether the $classifier 1$ is a better classifier than the $classifier 2$ on the base dataset. What t-test is appropriate to use?

  • $\begingroup$ Did you test the classifiers on the same samples? i.e., sample1, c1(sample1), c2(sample1)? Or did you use different samples for each classifier? $\endgroup$ Commented Dec 13, 2012 at 20:59
  • $\begingroup$ Paired t-test would be appropriate in this scenario. $\endgroup$
    – GEL
    Commented Dec 13, 2012 at 22:29
  • 1
    $\begingroup$ @lewellen: accuracy is a proportion: t-tests are usually not appropriate. $\endgroup$ Commented Dec 14, 2012 at 0:00
  • 3
    $\begingroup$ @JohnMoeller: "difference of proportions" would be a search term, independent or dependent we don't know yet. If it's paired: McNemar's test. I'm guessing that t-test means rather small sample size, so possibly normal approximation is not a good idea. I'd go for Statistical Methods for Rates and Proportions to look up details. $\endgroup$ Commented Dec 14, 2012 at 0:43
  • 2
    $\begingroup$ @JohnMoeller: I'm saying that each accuracy is a proportion. If you want to compare them, use methods for "difference of proportions". I expanded this into an answer to prevent endless comments. $\endgroup$ Commented Dec 14, 2012 at 1:31

2 Answers 2


A review and critique of some t-test approaches is given in Choosing between two learning algorithms based on calibrated tests, Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms, and On Comparing Classifiers: Pitfalls to Avoid and a Recommended Approach

  • $\begingroup$ Dietterich says: "The binomial distribution can be well-approximated by a normal distribution for reasonable values of $n$." So far, you did not tell us that you have reasonable $n$. @JohnMoeller's 30 cases are IMHO rather few for the normal approximation (at least without any knowledge about $p_1$ and $p_2$). $\endgroup$ Commented Dec 14, 2012 at 16:11
  • $\begingroup$ I have at-least 4000 records per class available in the base dataset, therefore the sample I select can be anything less than this. The biggest drawback with the difference of proportions tests are that they ignore "internal variation of the learning algorithm". I think this is important for a classifier such a a neural network, which I am using. $\endgroup$
    – entropy
    Commented Dec 14, 2012 at 17:36
  • $\begingroup$ well, that is a completely different situation from what JohnMoeller chimed in with. If you mean instability of the model by "internal variation": you can measure this. I'll update my answer. $\endgroup$ Commented Dec 14, 2012 at 17:49
  • $\begingroup$ To clarify, 30 is the number of times I select test/train partition sets, not the number of test points I select. $\endgroup$ Commented Dec 14, 2012 at 19:43
  • $\begingroup$ @JohnMoeller: sorry, I completely misunderstood that (coming from a field where "a sample" is a physical specimen of some sort). $\endgroup$ Commented Dec 15, 2012 at 20:13

I don't have the Fleiss book at hand, so all this is IIRC.

Answering @JohnMoeller's question in the comments for the moment: the original question is IMHO unanswerable as it is.

So suppose that I have 30 samples, and I test c1 and c2 on each sample, and record the accuracy for each on each sample.

doing this, you end up with a 2 x 2 contingency table giving classifier 1 correct/wrong against classifier 2 correct/wrong. Which is the starting point for McNemar's test. So this is for a paired comparison, which is more powerful than comparing "independent" proportions (which are not completely independent if they come from drawing randomly from the same finite sample).

I cannot look up McNemar's "small print" right now, but 30 samples is not much. So you may even have to switch from McNemar's to Fisher's exact test [or something else] which calculates the binomial probabilities.

Means of proportions:
It doesn't matter whether you test one and the same classifier 10x with 10 test cases or once with all those 100 cases (the 2 x 2 table just counts all test cases).

If the 10 estimates of accuracy for each classifier in the original question are obtained by random hold out or 10-fold cross validation or 10x out-of-bootstrap, the assumption is usually that the 10 surrogate models calculated for each classifier are equivalent (= have the same accuracy), so test results can be pooled*. For 10-fold cross validation you then assume that the test sample size equals the total number of test samples. For the other methods I'm not so sure: you may test the same case more than once. Depending on the data/problem/application, this doesn't amount to as much information as testing a new case.

*If the surrogate models are unstable, this assumption breaks down. But you can measure this: Do iterated $k$-fold cross validation. Each complete run gives one prediction for each case. So if you compare the predictions for the same test case over a number of different surrogate models, you can measure the variance caused by exchanging some of the training data. This variance is in addition to the variance due to the finite total sample size.

Put your iterated CV results into a "correct classification matrix" with each row corresponding to one case and each column to one of the surrogate models. Now the variance along the rows (removing all empty elements) is solely due to instability in the surrogate models. The variance in the columns is due to the finite number of cases you used for testing of this surrogate model. Say, you have $k$ correct predicitions out of $n$ tested cases in a column. The point estimate for the accuracy is $\hat p = \frac{k}{n}$, it is subject to variance $\sigma^2 (\hat p) = \sigma^2 (\frac{k}{n}) = \frac{p (1 - p)}{n}$.
Check whether the variance due to instability is large or small compared to the variance due to finite test sample size.

  • $\begingroup$ Ah, ok. It's the last bit that clears things up, at least for me. Thanks. $\endgroup$ Commented Dec 14, 2012 at 2:02
  • $\begingroup$ Thanks for the response. Just not clear on the procedure to follow. You say preform 10-fold cross validation on a single dataset. Measure the accuracy on the hold out sample, that is compute a 2x2 confusion matrix. Add up the ten 2x2 confusion matrices. Preform the McNemar's test on the aggregated 2x2 confusion matrix. $\endgroup$
    – entropy
    Commented Dec 14, 2012 at 14:41
  • $\begingroup$ @entropy: 1. The 2x2 contingency table is not the confusion matrix. 2. new sample each time vs. testing both classifiers on the same testing data: paired tests are more powerful (and possible here). See the updated answer. $\endgroup$ Commented Dec 14, 2012 at 15:56
  • $\begingroup$ Sorry for that, yes, contingency table. Am I correct to say that McNemar's test translate directly to a multi-class problem also. $\endgroup$
    – entropy
    Commented Dec 14, 2012 at 17:43
  • $\begingroup$ @cbeleites thanks so much for the response!!! I think you have now answered my questions exactly. However, i still don’t understand the exact procedure to follow. Would you mind just elaborating on the last paragraph. $\endgroup$
    – entropy
    Commented Dec 14, 2012 at 18:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.