Following on from an earlier question, I've got another question about comparing machine learning classifiers on various datasets. From the response to that question, and also to this paper I understand that good practice is to compare multiple classifiers on multiple datasets by using a Friedman test with Nemenyi post-hoc pairwise comparisons.
However this approach is only appropriate for situations where you have one accuracy value per classifier model on each dataset, i.e.
Dataset Alg1 Alg2 Alg3 A 0.85 0.72 0.79 B 0.92 0.85 0.89 C 0.78 0.82 0.81
In these situations it's a repeated measures study with the Datasets being the subjects. However, I generally use stochastic algorithms and repeat each run multiple times, generally 30 or so. So for every algorithm, on each dataset I run 10-fold cross-validation, take the cross-validated mean score as that run's accuracy and repeat that 30 times.
So my dataset of results looks more like (with 3 runs per algorithm):
Run Dataset Alg1 Alg2 Alg3 1 A 0.75 0.82 0.83 2 A 0.81 0.92 0.75 3 A 0.91 0.81 0.64 1 B 0.77 0.89 0.71 2 B 0.98 0.87 0.83 3 B 0.62 0.71 0.68 1 C 0.82 0.87 0.83 2 C 0.91 0.82 0.71 3 C 0.81 0.85 0.83
What test would be appropriate to compare these results on? One method I've come up with is to simply average the runs on each dataset to get a table looking like the first example, upon which the Friedman test can simply be run. Another more naive method would be to consider the datasets separately, and take the runs as samples. Then you could do 3 pairwise Wilcoxon tests on each dataset (accounting for multiple comparisons) to see on each dataset where the statistically significant differences lie.
Are either of these approaches valid, and if not, what is the most appropriate way to analyse this data?