I have 3 models from which, for each model, I train a classifier and then evaluate it, currently using stratified 10-fold cross validation and then take the mean accuracy ratio of these from each fold. I currently measure the mean accuracy and standard deviation. For each model, I perform the same procedure and compare by myself the mean accuracies.
I use this experimental setup over 3 datasets, so that I can validate if my models are good for general use.
I, by now, want to use statistical test to compare, for a certain dataset, and for 2 different models, if the resulting classifiers produce equivalent results or not. It seems that, by only comparing mean accuracy there is no guaranty I have statistical significance or not.
I have been told to use wilcoxon signed-rank test, or Poisson binomial test. I still don't know exactly how to use wilcoxon signed-rank test, for example. Should I compare the 10-fold results from model 1, vs 10-fold results from model 2? Or should I take, for instance, the mean from each model and then use the statistical test? Or, maybe, should I use other metrics than accuracy, such as F-measure?
Do you have statistical test methods to recommend and why, as well as instructions on how to use then? Please just remind that my case compares 3 models, and I use 3 datasets as the basis for comparison (experiments on each dataset are performed separately).