When comparing the performance of two classifiers over a single domain, in the context of a classification problem in machine learning, it is common to use a paired t-test, using the 10 average results from 10x10-fold cross-validation as measurements, where the folds at each iteration are the same for the two classifiers.
An obvious yet incorrect generalization of this t-test to the case where there are multiple classifiers would be to take all pairs, or just the interesting pairs if we only want to compare 1 classifier to all the others, and to work from there. The problem, however, is that the results of these multiple t-tests are not independent from one another.
I've heard of the ANOVA as a generalization of the t-test to situations with more than 2 groups to compare. I thought it was the solution to this problem, but then I read more and stumbled upon the repeated measures ANOVA, which seems to be even closer to what I'm looking for.
Could anyone confirm (or invalidate) that a repeated measures ANOVA is indeed what should be used in this situation? What is the difference between a regular ANOVA and the rANOVA in this case?