Following on from an earlier question, I've got another question about comparing machine learning classifiers on various datasets. From the response to that question, and also to this paper I understand that good practice is to compare multiple classifiers on multiple datasets by using a Friedman test with Nemenyi post-hoc pairwise comparisons.

However this approach is only appropriate for situations where you have one accuracy value per classifier model on each dataset, i.e.

Dataset     Alg1    Alg2    Alg3
   A        0.85    0.72    0.79
   B        0.92    0.85    0.89
   C        0.78    0.82    0.81

In these situations it's a repeated measures study with the Datasets being the subjects. However, I generally use stochastic algorithms and repeat each run multiple times, generally 30 or so. So for every algorithm, on each dataset I run 10-fold cross-validation, take the cross-validated mean score as that run's accuracy and repeat that 30 times.

So my dataset of results looks more like (with 3 runs per algorithm):

Run    Dataset      Alg1    Alg2    Alg3
  1       A         0.75    0.82    0.83
  2       A         0.81    0.92    0.75
  3       A         0.91    0.81    0.64
  1       B         0.77    0.89    0.71
  2       B         0.98    0.87    0.83
  3       B         0.62    0.71    0.68
  1       C         0.82    0.87    0.83 
  2       C         0.91    0.82    0.71
  3       C         0.81    0.85    0.83

What test would be appropriate to compare these results on? One method I've come up with is to simply average the runs on each dataset to get a table looking like the first example, upon which the Friedman test can simply be run. Another more naive method would be to consider the datasets separately, and take the runs as samples. Then you could do 3 pairwise Wilcoxon tests on each dataset (accounting for multiple comparisons) to see on each dataset where the statistically significant differences lie.

Are either of these approaches valid, and if not, what is the most appropriate way to analyse this data?

  • $\begingroup$ Hello again Stuart, what you are proposing is a replicated complete block design which I am not as familiar with. Technically, there is an extended friedman test with the prentice.test function of the muStat package. However, I have read that it may not be best to extend Friedman to these circumstances. I suggest updating your question title to more accurately address the statistical question (i.e. nonparametric repeated measures analysis of replicated block data). I suspect you may want to explore mixed models. $\endgroup$ – cdeterman Dec 16 '14 at 20:56
  • $\begingroup$ Hi again, thanks for the comment. I was wondering if there was a more accurate term to describe this sort of data, I'll have a read upon replicated block design and see if that gets me anyway. $\endgroup$ – Stuart Lacy Dec 18 '14 at 11:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.