Statistical Significance of multiple classifiers by using p-value

Question

I have 675 samples, and used 7 machine learning algorithms with 10-fold cross-validation for prediction in a classification problem. Let's say the following table is the accuracy result of each algorithm.

My supervisor asked me to perform a p-value test of the results. Is it possible to get the p-value test of these data to make a claim that the results are statistically significant? Or it is an irrelevant test for such a context?

If the answer is yes p-value is appropriate, then how can I do it? Is it possible with Python or Excel etc.?

Answer 1

This is common practice but quite controversial among statisticians (Dietterich 1998) (Kohavi 1995)

For a t-test you need 30 or more samples if you cannot assume a normal distribution (which you really cannot). The common approach to this is to have 3 repetitions of a 10 fold CV. The folds should be shuffled by a random generator, but the same random folds should be used with each algorithm. Use a one sample t-test on the difference of performance.

T-tests also assume i.i.d. samples. The independence of the samples is not given in CV. This problem can be avoided by using very different non-parametric tests. Or you could use corrected resampled t-tests that correct precisely for this non independence in repeated cross validation settings. They are less powerful than normal t-tests, but arguably more so than the non parametric tests linked above. Weka uses these tests per default for example.

PS. If you compare 7 algorithms pairwise, you also need a Bonferroni adjustment or something similar to your p-value.

Answer 2

I agree with Ramalho that it is a good idea to use statistical hypothesis testing for interpretation of model comparison.

Here are some points, that should give you a start:

• You can and should set up your performance measurements so that all algorithms use exactly the same splits. This gives you a paired test setup. Paired tests are more powerful than unpaired tests.

• Yes, you should also do iterations/repetitions of the cross validation, because that allows you to compare the differences you observe between algorithms to model (in)stability.

• However, keep in mind that your test sample size stays the number of different cases you have available and does not increase with repeating the cross valdiation.

• Also you should know that 0/1 loss and related figures of merit (such as accuracy) are quite bad behaved from a statistical point of view: they have high variance and thus need large sample sizes to become precise. So-called proper scoring rules would be a better choice, but you have to weight into the decision whether your audience would accept that.

• To give you a starting point: with 0/1 loss, a (single) paired comparison can be done with McNemar's test. As @user7019377 says, for multiple comparisons you need to do appropriate corrections.

• Even better, though would be to limit the number of comparisons you actually do. So when setting up the test, think hard how to formulate your hypotheses. You may want to show that your stratified dummy is not significantly different from 50 % and the random dummy not significantly different from 1/nclasses (if I'm correctly guessing the situation). Also, glancing at the results you got for the real classifiers, you may not need multiple tests there but a single test showing that no algorithm predicts significantly different from any other may be sufficient. Unfortunately, I don't know whether there's something like an ANOVA for binomial data. However, I'd go for a proper scoring rule and have a look whether ANOVA would be a suitable choice.

  • However, here's something that may provide you a shortcut that gives you all that you need for your interpretation:
  • The tests for the dummy classifiers are pretty straightforward textbook things.

  • For the real classifiers, a quick calculation of binomial confidence intervals reveals that 2-sided 95% confidence intervals of the accuracies you report for 675 test cases enclose all of the other classifiers (e.g. 610 of 675 => c.i. from about 87.9% to 92.4%)

  • looking at the "best" possible results with McNemar's test between KNN with 599 correct and SVM with 610 correct => that would be if the SVN gets all those cases correct that KNN got correct as well, plus additional 11 cases. This gets an (uncorrected) p-value of 0.0025. I.e. after whatever multiple testing correction you prefer, it won't be significant at the p = 0.05 level. Also, one single case that KNN got correct but SVM didn't catapults the p-value for McNemar's test between SVM and kNN above the p = 0.05 value.

  Even if an ANOVA-analogue for McNemar's test found that not all 5 real classifiers are equivalent, you don't have enough test cases to conclude which is better!

  => For all practical purposes, I don't see any significant (also and particularly in the every-day sense of the word!) difference between the results for the 5 classifiers you used here (This is based on the assumption that the results were gotten in a paired setup - if the experiment wasn't paired as described above, it is far from siginificant but then things may look different for a paired experiment)

---

• Instead of the binomial tests, also rank-based tests are used.

• @air mentions the correlation between the cross validation surrogate models. Whether this is a problem or an advantage depends on what you actually want to compare. If you are looking from an application point of view and want to answer the question what performance to expect for the model trained on this particular dataset using the training algorithm - then the high correlation is what you want. Actually, you assume equality! The correlation is problematic if you want to treat your resampled data sets as approximations to drawing new training data sets for this application (i.e. want to compare achievable performance for different training algorithms on similar tasks).
  In the Dietterich1998 paper that corresponds to the question whether you analyze a classier or an algorithm.

Answer 3

You might consider adding the best naive classifier you can think of, assign classes based on some simple historical average, or assign randomly based on expected distribution. (Perhaps that's the 'random dummy'.)

Then do a chi-square test on the confusion matrix of each test vs. the naive classifier to determine the likelihood of the improvement being due to chance.

Answer 4

Yes you can apply statistical hypothesis testing.

You evaluate each classifier with cross-validation (like you did) but instead store the (accuracy/other metric) of that classifier during each fold. After you have done that you can use hypothesis testing to test if the mean acc of that classifier is significantly better or worst than the mean acc of a second classifier. You can even rank them this way.

Regarding this being a good or bad idea.. I find it is a good idea to compare the same classifier while using different sets of features or transformations.. but comparing different models? I don't know. You might be better off combining the classifiers through an ensemble.

Repeating cross-validation is also a good idea if you pursuit the method I described. Specially if you want to achieve more precise/stable results (crossvalidation usually has high variance).

Yes, I did it before... on python. It's really straightforward.