What's the correct way to test the significance of classification results There are many situations where you may train several different classifiers, or use several different feature extraction methods. In the literature authors often give the mean classification error over a set of random splits of the data (i.e. after a doubly nested cross-validation), and sometimes give variances on the error over the splits as well. However this on its own is not enough to say that one classifier is significantly better than another. I've seen many different approaches to this - using Chi-squared tests, t-test, ANOVA with post-hoc testing etc. 
What method should be used to determine statistical significance? Underlying that question is: What assumptions should we make about the distribution of classification scores?
 A: Since distribution of classification errors is a binary distribution (either there is misclassifcation or there is none) --- I'd say that using Chi-squared is not sensible. 
Also only comparing efficiences of classifiers that work on the same datasets is sensible --- 
'No free lunch theorem' states that all models have the same average efficiency over all datasets, so that which model will appear better will depend only on what datasets were choosen to train them http://en.wikipedia.org/wiki/No_free_lunch_in_search_and_optimization. 
If you are comparing efficiency of models A and B over dataset D i think that average efficiency + mean is enough to make a choice. 
Moreover if one has many models that have resonable efficiency (and are lineary independent of each other) I'd rather build ensemble model than just choose best model. 
A: I recommend the paper by Tom Dietterich titled "Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms".  Here's the paper's profile on CiteSeer.  From the abstract: "This paper reviews five approximate statistical tests for determining whether one learning algorithm out-performs another on a particular learning task. These tests are compared experimentally to determine their probability of incorrectly detecting a difference when no difference exists (type I error). ...  McNemar's test, is shown to have low Type I error. ..."
A: IMHO there shouldn't be any different between distribution of scores to  distribution of any other type of data. so basically all you have to check is whether your data is distributed normally or not see here. Moreover, There are great books that deal thoroughly with this question see here (i.e. in short: they all test whether the outcome of two classifier is significantly different.. and if they do, they can be combined into one - ensemble model)
A: There is no single test that is appropriate for all situations; I can recommend the book "Evaluating Learning Algorithms" by Nathalie Japkowicz and Mohak Shah, Cambridge University Press, 2011.  The fact that a book of almost 400 pages can be written on this topic suggests it isn't a straight-forward issue.  I have often found that there isn't a test that really suits the needs of my study, so it is important to have a good grasp of the advantages and disadvantages of whatever method is eventually used.
A common problem is that for large datasets a statistically significant difference may be obtained with an effect size that is of no practical significance.  
A: In addition to @jb.'s excellent answer, let me add that you can use McNemar's test on the same test set to determine if one classifier is significantly better than the other. This will only work for classification problems (what McNemar's original work call a "dichotomous trait") meaning that the classifiers either get it right or wrong, no space in the middle. 
