There are many situations where you may train several different classifiers, or use several different feature extraction methods. In the literature authors often give the mean classification error over a set of random splits of the data (i.e. after a doubly nested cross-validation), and sometimes give variances on the error over the splits as well. However this on its own is not enough to say that one classifier is significantly better than another. I've seen many different approaches to this - using Chi-squared tests, t-test, ANOVA with post-hoc testing etc.
What method should be used to determine statistical significance? Underlying that question is: What assumptions should we make about the distribution of classification scores?