I had a classification problem. I had 675 samples, and used with 7 machine learning algorithms with 10 cross-validation for prediction. Lets say the following table is the accuracy result of each algorithm.

My supervisor asked me to perform a p-value test of the results, I have searched and read some, but I am not confident if such a test is appropriate. So, I wonder, is it possible (and makes sense) to get the p-value test of these data to claim result are statistically significant? Or it is a irrelevant test for such a context?

If the answer is yes p-value is appropriate, then how can I do it, is it possible with python or excel etc...

Many thanks


  • 1
    $\begingroup$ What exactly your supervisor would like you to test? What is the null hypothesis to be tested? I can't see any meaningful way of applying hypothesis test to classification results... $\endgroup$ – Tim Oct 16 '16 at 17:50
  • $\begingroup$ Thank you everyone who replied/commented the post. We are searching an answer for, "are the 7 row values statistically different? if yes to what p-value? 0.01 0.05 or what? We expect this will tell us with statistical rigour result is when used for prediction by different machine learning algorithms." $\endgroup$ – Memin Oct 16 '16 at 22:36

This is common practice but quite controversial among statisticians (Dietterich 1998) (Kohavi 1995)

For a t-test you need 30 or more samples if you cannot assume a normal distribution (which you really cannot). The common approach to this is to have 3 repetitions of a 10 fold CV. The folds should be shuffled by a random generator, but the same random folds should be used with each algorithm. Use a one sample t-test on the difference of performance.

T-tests also assume i.i.d. samples. The independence of the samples is not given in CV. This problem can be avoided by using very different non-parametric tests. Or you could use corrected resampled t-tests that correct precisely for this non independence in repeated cross validation settings. They are less powerful than normal t-tests, but arguably more so than the non parametric tests linked above. Weka uses these tests per default for example.

PS. If you compare 7 algorithms pairwise, you also need a Bonferroni adjustment or something similar to your p-value.

| cite | improve this answer | |
  • $\begingroup$ Thank you for replying, to be honest I am totally new to these statistical terms and calculation and quite missed how to implement these things correctly. So I will ask a precise question. If get the accuracy scores of each fold for each classifier. Then if I use One-Way ANOVA (since we have multiple classes) as suggested here "hamelg.blogspot.co.uk/2015/11/…" and calculate stats.f_oneway to get p-value, will that reflect the statistical significance of the results? I am sorry for direct question, but I need to be sure if I am doing it correctly. $\endgroup$ – Memin Oct 16 '16 at 22:41
  • $\begingroup$ You are not really interested in the ANOVA itself. Your data will routinely violate the sphericity assumption. However, it may be the case that your software provides only a package deal of ANOVA + post-hoc tests. (SPSS does this I think.) Those post hoc tests are of interest here. They are independent from the ANOVA, so you can safely ignore it and look only at the post-hoc tests. $\endgroup$ – David Ernst Oct 16 '16 at 23:15
  • $\begingroup$ The groups in the one way ANOVA represent the algorithms, not the classes. Classes need to be dealt with before handing data to the ANOVA + post hoc by using a performance metric that can handle multiple classes (accuracy, cohens kappa or multi-class F-measure for example) $\endgroup$ – David Ernst Oct 16 '16 at 23:17
  • $\begingroup$ @user7019377 +1 for mentioning corrected resampled t-test. I have heard of it before but haven't managed to get further details than Weka uses it. What exactly is this test? Is it implemented in any other software like R? Does it require input from each fold? $\endgroup$ – DataD'oh Sep 3 '17 at 20:33
  • $\begingroup$ It just computes the variance of the sample mean point estimator differently (wider) to correct for dependent samples. You can easily implement it in any software with a couple of lines of code. Section 3.3 in the paper I linked shows the equation. $\endgroup$ – David Ernst Sep 3 '17 at 20:48

I agree with Ramalho that it is a good idea to use statistical hypothesis testing for interpretation of model comparison.

Here are some points, that should give you a start:

  • You can and should set up your performance measurements so that all algorithms use exactly the same splits. This gives you a paired test setup. Paired tests are more powerful than unpaired tests.

  • Yes, you should also do iterations/repetitions of the cross validation, because that allows you to compare the differences you observe between algorithms to model (in)stability.

  • However, keep in mind that your test sample size stays the number of different cases you have available and does not increase with repeating the cross valdiation.

  • Also you should know that 0/1 loss and related figures of merit (such as accuracy) are quite bad behaved from a statistical point of view: they have high variance and thus need large sample sizes to become precise. So-called proper scoring rules would be a better choice, but you have to weight into the decision whether your audience would accept that.

  • To give you a starting point: with 0/1 loss, a (single) paired comparison can be done with McNemar's test. As @user7019377 says, for multiple comparisons you need to do appropriate corrections.

  • Even better, though would be to limit the number of comparisons you actually do. So when setting up the test, think hard how to formulate your hypotheses. You may want to show that your stratified dummy is not significantly different from 50 % and the random dummy not significantly different from 1/nclasses (if I'm correctly guessing the situation). Also, glancing at the results you got for the real classifiers, you may not need multiple tests there but a single test showing that no algorithm predicts significantly different from any other may be sufficient. Unfortunately, I don't know whether there's something like an ANOVA for binomial data. However, I'd go for a proper scoring rule and have a look whether ANOVA would be a suitable choice.

    • However, here's something that may provide you a shortcut that gives you all that you need for your interpretation:
    • The tests for the dummy classifiers are pretty straightforward textbook things.
    • For the real classifiers, a quick calculation of binomial confidence intervals reveals that 2-sided 95% confidence intervals of the accuracies you report for 675 test cases enclose all of the other classifiers (e.g. 610 of 675 => c.i. from about 87.9% to 92.4%)

    • looking at the "best" possible results with McNemar's test between KNN with 599 correct and SVM with 610 correct => that would be if the SVN gets all those cases correct that KNN got correct as well, plus additional 11 cases. This gets an (uncorrected) p-value of 0.0025. I.e. after whatever multiple testing correction you prefer, it won't be significant at the p = 0.05 level. Also, one single case that KNN got correct but SVM didn't catapults the p-value for McNemar's test between SVM and kNN above the p = 0.05 value.

    Even if an ANOVA-analogue for McNemar's test found that not all 5 real classifiers are equivalent, you don't have enough test cases to conclude which is better!

    => For all practical purposes, I don't see any significant (also and particularly in the every-day sense of the word!) difference between the results for the 5 classifiers you used here (This is based on the assumption that the results were gotten in a paired setup - if the experiment wasn't paired as described above, it is far from siginificant but then things may look different for a paired experiment)

  • Instead of the binomial tests, also rank-based tests are used.

  • @air mentions the correlation between the cross validation surrogate models. Whether this is a problem or an advantage depends on what you actually want to compare. If you are looking from an application point of view and want to answer the question what performance to expect for the model trained on this particular dataset using the training algorithm - then the high correlation is what you want. Actually, you assume equality! The correlation is problematic if you want to treat your resampled data sets as approximations to drawing new training data sets for this application (i.e. want to compare achievable performance for different training algorithms on similar tasks).
    In the Dietterich1998 paper that corresponds to the question whether you analyze a classier or an algorithm.

| cite | improve this answer | |

You might consider adding the best naive classifier you can think of, assign classes based on some simple historical average, or assign randomly based on expected distribution. (Perhaps that's the 'random dummy'.)

Then do a chi-square test on the confusion matrix of each test vs. the naive classifier to determine the likelihood of the improvement being due to chance.

| cite | improve this answer | |

Yes you can apply statistical hypothesis testing.

You evaluate each classifier with cross-validation (like you did) but instead store the (accuracy/other metric) of that classifier during each fold. After you have done that you can use hypothesis testing to test if the mean acc of that classifier is significantly better or worst than the mean acc of a second classifier. You can even rank them this way.

Regarding this being a good or bad idea.. I find it is a good idea to compare the same classifier while using different sets of features or transformations.. but comparing different models? I don't know. You might be better off combining the classifiers through an ensemble.

Repeating cross-validation is also a good idea if you pursuit the method I described. Specially if you want to achieve more precise/stable results (crossvalidation usually has high variance).

Yes, I did it before... on python. It's really straightforward.

| cite | improve this answer | |
  • $\begingroup$ Uhm unless you can provide a reference, I really don't think it's that straight-forward. Correlations across folds in cross-validation (because the training folds always overlap) makes variance estimation and consequent testing a very difficult problem, as far as I am aware. $\endgroup$ – air Oct 16 '16 at 9:35
  • $\begingroup$ I dont know if Im following your concern, what I meant is to use the training folds as training data and test it according to a desired metric. This will give you a sample of the performance of your model and hyperparameters. Training folds overlap yes and that can lead to violation of independence yes, but you apply the same procedure to all classifiers, they will be subject to this less ideal scheme. Sorry I dont have a paper on this. $\endgroup$ – Ramalho Oct 16 '16 at 9:44
  • $\begingroup$ Think of the problem as the problem of testing if two classes of students that were subject to different teaching methods show significative differences. In practice here will always some overlap, will that be limitation? In practice, probably not. $\endgroup$ – Ramalho Oct 16 '16 at 10:01
  • $\begingroup$ Thank you Ramalho for replying, do you think the comment I have for @user7019377 post, will be right approach to get p-values. $\endgroup$ – Memin Oct 16 '16 at 22:43
  • $\begingroup$ If found @cbeleites explanation to be much more comprehensive, I think I have learned a thing or two! :) I would advise you to follow it. To my experience I never had the problem of non-normality, but sure it can happen, in which case I would advise you to lookup for a non parametric test. Think of it, the problem of non normality will surely be more preeminent if you use higher complexity classifiers, those will very likely be more sensitive to the folds and in turn bring higher variance to your sample of statistics. Pick your statistic/metric wisely. $\endgroup$ – Ramalho Oct 17 '16 at 17:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.