I have 3 models from which, for each model, I train a classifier and then evaluate it, currently using stratified 10-fold cross validation and then take the mean accuracy ratio of these from each fold. I currently measure the mean accuracy and standard deviation. For each model, I perform the same procedure and compare by myself the mean accuracies.

I use this experimental setup over 3 datasets, so that I can validate if my models are good for general use.

I, by now, want to use statistical test to compare, for a certain dataset, and for 2 different models, if the resulting classifiers produce equivalent results or not. It seems that, by only comparing mean accuracy there is no guaranty I have statistical significance or not.

I have been told to use wilcoxon signed-rank test, or Poisson binomial test. I still don't know exactly how to use wilcoxon signed-rank test, for example. Should I compare the 10-fold results from model 1, vs 10-fold results from model 2? Or should I take, for instance, the mean from each model and then use the statistical test? Or, maybe, should I use other metrics than accuracy, such as F-measure?

Do you have statistical test methods to recommend and why, as well as instructions on how to use then? Please just remind that my case compares 3 models, and I use 3 datasets as the basis for comparison (experiments on each dataset are performed separately).


You should run your 10 fold cross validation 10 to 100 times and use the cross validation result from each of those experiments, for example if you run 100 cross validations your sample should have a 100 observations. If the assumptioms hold you could even use simple hypothesis testing (pairwise) and confidence intervals to check if the model performance is significantly better. Regarding using other metrics, each metric tells you a different story about your model, so it might be benefic to run experiments over more than one.. As a matter of fact, it is common.

| cite | improve this answer | |
  • $\begingroup$ sadly I can't afford running cross validation from 10 to 100 times as you suggested: the scenario take a lot a time. You mentioned I should do it this way, but I still need valid references for both your suggestions and those I mentioned in the question. $\endgroup$ – Ícaro Dourado Sep 18 '15 at 0:19
  • 1
    $\begingroup$ It is not appropriate to run a statistical test on cross-validation estimates when the statistical test assumes observations are independent. The multiple validates are correlated because they use overlapping data. $\endgroup$ – Frank Harrell Apr 28 '18 at 11:49
  • 1
    $\begingroup$ You did not describe the ultimate goal, and whether there are probabilities of outcomes "underneath the hood". If there are, then you are using improper scoring rules that lose a lot of information and are arbitrary. $\endgroup$ – Frank Harrell May 29 '18 at 11:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.