24
$\begingroup$

There are many situations where you may train several different classifiers, or use several different feature extraction methods. In the literature authors often give the mean classification error over a set of random splits of the data (i.e. after a doubly nested cross-validation), and sometimes give variances on the error over the splits as well. However this on its own is not enough to say that one classifier is significantly better than another. I've seen many different approaches to this - using Chi-squared tests, t-test, ANOVA with post-hoc testing etc.

What method should be used to determine statistical significance? Underlying that question is: What assumptions should we make about the distribution of classification scores?

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ Could you post example papers with: "I've seen many different approaches to this - using Chi-squared tests, t-test, ANOVA with post-hoc testing etc."? I'm really interested in that. $\endgroup$
    – jb.
    Feb 8, 2012 at 17:14
  • 1
    $\begingroup$ @jb take a look on this one: cmpe.boun.edu.tr/~ethem/i2ml/slides/v1-1/i2ml-chap14-v1-1.pdf $\endgroup$
    – Dov
    Feb 8, 2012 at 17:19

5 Answers 5

Reset to default
10
$\begingroup$

In addition to @jb.'s excellent answer, let me add that you can use McNemar's test on the same test set to determine if one classifier is significantly better than the other. This will only work for classification problems (what McNemar's original work call a "dichotomous trait") meaning that the classifiers either get it right or wrong, no space in the middle.

Improve this answer
$\endgroup$
1
  • $\begingroup$ What about in the scenario when the classifier can pass? As in it says it does not know. Can you still use McNemar's test then? $\endgroup$
    – S0rin
    May 12, 2014 at 15:25
5
$\begingroup$

Since distribution of classification errors is a binary distribution (either there is misclassifcation or there is none) --- I'd say that using Chi-squared is not sensible.

Also only comparing efficiences of classifiers that work on the same datasets is sensible --- 'No free lunch theorem' states that all models have the same average efficiency over all datasets, so that which model will appear better will depend only on what datasets were choosen to train them http://en.wikipedia.org/wiki/No_free_lunch_in_search_and_optimization.

If you are comparing efficiency of models A and B over dataset D i think that average efficiency + mean is enough to make a choice.

Moreover if one has many models that have resonable efficiency (and are lineary independent of each other) I'd rather build ensemble model than just choose best model.

Improve this answer
$\endgroup$
4
  • $\begingroup$ But for a single classifier you end up with a set of scores (e.g. MSE over 100 splits), which could be in the range [0,1] for example. I think it would be far too expensive to take the results of every single run and analyse them. $\endgroup$
    – tdc
    Feb 9, 2012 at 9:51
  • $\begingroup$ Yes. But in this case mean + stddev is enough to test whether one is significantly better than the other, just like with any other measurement. $\endgroup$
    – jb.
    Feb 9, 2012 at 10:19
  • 2
    $\begingroup$ I'm not so sure. Mean & stddev assumes Gaussianity for a start, and secondly this doesn't take into account how many comparisons are being done (e.g. Bonferroni correction might be needed) $\endgroup$
    – tdc
    Feb 9, 2012 at 10:53
  • 1
    $\begingroup$ It is the same in basic measuremnt theory. Lets assume we have a micrometer and we want to check whether two rods have the same diamater, we take 100 measurements of both rods and check whether mean + stddev overlap. In both cases (rod measurrement and model metic) we just assume gaussian distribution of results, only sensible argument is Central limit theorem. $\endgroup$
    – jb.
    Feb 9, 2012 at 11:13
3
$\begingroup$

I recommend the paper by Tom Dietterich titled "Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms". Here's the paper's profile on CiteSeer. From the abstract: "This paper reviews five approximate statistical tests for determining whether one learning algorithm out-performs another on a particular learning task. These tests are compared experimentally to determine their probability of incorrectly detecting a difference when no difference exists (type I error). ... McNemar's test, is shown to have low Type I error. ..."

Improve this answer
$\endgroup$
2
$\begingroup$

IMHO there shouldn't be any different between distribution of scores to distribution of any other type of data. so basically all you have to check is whether your data is distributed normally or not see here. Moreover, There are great books that deal thoroughly with this question see here (i.e. in short: they all test whether the outcome of two classifier is significantly different.. and if they do, they can be combined into one - ensemble model)

Improve this answer
$\endgroup$
6
  • $\begingroup$ I think they're very likely not to be distributed normally. In the usual case the scores will be positive and skewed towards one end of the range (1 or 0 depending if you are using accuracy or error as the measure). $\endgroup$
    – tdc
    Feb 9, 2012 at 9:52
  • $\begingroup$ @tdc: this case distribution of function (number of misclassifications) -> (number of models with this count of misclassifcations) would be often IMHO similar poisson disrtibution. $\endgroup$
    – jb.
    Feb 9, 2012 at 10:27
  • $\begingroup$ @Dov: Testing which model is significantly better (that is the OP question) and testin if they are different is a quite different thing. $\endgroup$
    – jb.
    Feb 9, 2012 at 10:29
  • $\begingroup$ @jb. thanks. but i said significantly different not better... $\endgroup$
    – Dov
    Feb 9, 2012 at 15:15
  • $\begingroup$ @Dov your first link is broken - I can't tell where it's supposed to point. $\endgroup$
    – Tamzin Blake
    Feb 9, 2012 at 18:57
2
$\begingroup$

There is no single test that is appropriate for all situations; I can recommend the book "Evaluating Learning Algorithms" by Nathalie Japkowicz and Mohak Shah, Cambridge University Press, 2011. The fact that a book of almost 400 pages can be written on this topic suggests it isn't a straight-forward issue. I have often found that there isn't a test that really suits the needs of my study, so it is important to have a good grasp of the advantages and disadvantages of whatever method is eventually used.

A common problem is that for large datasets a statistically significant difference may be obtained with an effect size that is of no practical significance.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.