How to statistically compare the performance of machine learning classifiers?

Question

Based on estimated classification accuracy, I want to test whether one classifier is statistically better on a base set than another classifier . For each classifier, I select a training and testing sample randomly from the base set, train the model, and test the model. I do this ten times for each classifier. I therefore have ten estimate classification accuracy measurements for each classifier. How do I statistically test whether the $classifier 1$ is a better classifier than the $classifier 2$ on the base dataset. What t-test is appropriate to use?

Answer 1

A review and critique of some t-test approaches is given in Choosing between two learning algorithms based on calibrated tests, Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms, and On Comparing Classifiers: Pitfalls to Avoid and a Recommended Approach

Answer 2

I don't have the Fleiss book at hand, so all this is IIRC.

Answering @JohnMoeller's question in the comments for the moment: the original question is IMHO unanswerable as it is.

So suppose that I have 30 samples, and I test c1 and c2 on each sample, and record the accuracy for each on each sample.

doing this, you end up with a 2 x 2 contingency table giving classifier 1 correct/wrong against classifier 2 correct/wrong. Which is the starting point for McNemar's test. So this is for a paired comparison, which is more powerful than comparing "independent" proportions (which are not completely independent if they come from drawing randomly from the same finite sample).

I cannot look up McNemar's "small print" right now, but 30 samples is not much. So you may even have to switch from McNemar's to Fisher's exact test [or something else] which calculates the binomial probabilities.

---

Means of proportions:
It doesn't matter whether you test one and the same classifier 10x with 10 test cases or once with all those 100 cases (the 2 x 2 table just counts all test cases).

If the 10 estimates of accuracy for each classifier in the original question are obtained by random hold out or 10-fold cross validation or 10x out-of-bootstrap, the assumption is usually that the 10 surrogate models calculated for each classifier are equivalent (= have the same accuracy), so test results can be pooled*. For 10-fold cross validation you then assume that the test sample size equals the total number of test samples. For the other methods I'm not so sure: you may test the same case more than once. Depending on the data/problem/application, this doesn't amount to as much information as testing a new case.

*If the surrogate models are unstable, this assumption breaks down. But you can measure this: Do iterated $k$-fold cross validation. Each complete run gives one prediction for each case. So if you compare the predictions for the same test case over a number of different surrogate models, you can measure the variance caused by exchanging some of the training data. This variance is in addition to the variance due to the finite total sample size.

Put your iterated CV results into a "correct classification matrix" with each row corresponding to one case and each column to one of the surrogate models. Now the variance along the rows (removing all empty elements) is solely due to instability in the surrogate models. The variance in the columns is due to the finite number of cases you used for testing of this surrogate model. Say, you have $k$ correct predicitions out of $n$ tested cases in a column. The point estimate for the accuracy is $\hat p = \frac{k}{n}$, it is subject to variance $\sigma^2 (\hat p) = \sigma^2 (\frac{k}{n}) = \frac{p (1 - p)}{n}$.
Check whether the variance due to instability is large or small compared to the variance due to finite test sample size.