Statistical test for classification models

Question

I have 3 models from which, for each model, I train a classifier and then evaluate it, currently using stratified 10-fold cross validation and then take the mean accuracy ratio of these from each fold. I currently measure the mean accuracy and standard deviation. For each model, I perform the same procedure and compare by myself the mean accuracies.

I use this experimental setup over 3 datasets, so that I can validate if my models are good for general use.

I, by now, want to use statistical test to compare, for a certain dataset, and for 2 different models, if the resulting classifiers produce equivalent results or not. It seems that, by only comparing mean accuracy there is no guaranty I have statistical significance or not.

I have been told to use wilcoxon signed-rank test, or Poisson binomial test. I still don't know exactly how to use wilcoxon signed-rank test, for example. Should I compare the 10-fold results from model 1, vs 10-fold results from model 2? Or should I take, for instance, the mean from each model and then use the statistical test? Or, maybe, should I use other metrics than accuracy, such as F-measure?

Do you have statistical test methods to recommend and why, as well as instructions on how to use then? Please just remind that my case compares 3 models, and I use 3 datasets as the basis for comparison (experiments on each dataset are performed separately).

Answer 1

You should run your 10 fold cross validation 10 to 100 times and use the cross validation result from each of those experiments, for example if you run 100 cross validations your sample should have a 100 observations. If the assumptioms hold you could even use simple hypothesis testing (pairwise) and confidence intervals to check if the model performance is significantly better. Regarding using other metrics, each metric tells you a different story about your model, so it might be benefic to run experiments over more than one.. As a matter of fact, it is common.