I have trained a bunch of classifiers with nested cross-validation and now I'd like to perform some statistical tests to see if their difference in performance is meaningful. So I initially stumbled into this paper, which lead me to this, and so on. Looks like a simple paired t-test is not enough. I was thinking about performing McNemar's Test, 2x5CV F-Test possibly, corrected paired t-test definitely.

I'm quite confused about the methodology to conduct the tests, though. So please bear with me and tell me if my reasoning makes any sense.

So say I have 2 different models already trained, a dataset that I used to perform training/model selection/model evaluation with nested cross-validation, and another that I can use to perform statistical testings.

I would:

- use the fresh dataset (I suppose I can use the initial one too?);
- pick some number of repetitions (say 100);
- pick a test (say McNemar);
- random sample from the dataset 100 times (with or without replacement?);
- compute the performance of both models on that sample;
- compute the McNemar statistic from that same sample;
- report statistics of the 100 McNemar trials (mean, variance, CIs and whatnot).

Does this process make any sense to you? Any help or thoughts appreciated, thank you.

UPDATE Maybe I should point out that my classifiers are already trained (i. e. if you read the paper from Dietterich (first link), I'm in situation 3). This means that I cannot use the corrected paired T-test since it's a method to compare learning algorithms, not classifiers. Dietterich's paper states that I should be using McNemar's test to assess if the performance of the classifiers differs. Still, I have trouble understanding not the test itself (which is pretty simple), but the methodology in order to carry it on properly.


1 Answer 1


If the TWO classifiers are already trained (I assume from the first dataset) than you should run the TWO of them in the second dataset ONLY ONCE, and use the McNemar test. It will compute a p_value given the null hypothesis that the two classifiers are the same; if the p-value is low enough you can assume that it is unlikely that they are the same and thus their difference is statistically significant. If they are different, which one is the best? The one with higher accuracy on the data set you used in the statistical test.

It is very possible that you would not find the difference significant - the McNemar test does not depend on the size of the dataset, but only on the number of examples in the dataset that the two classifiers disagree! In this case, you will not be able to claim that the two classifiers are significantly different.

Finally, this is only valid for TWO classifiers. If you have more than 2 things are more complex. You will need to perform all pairwise 2-classifiers tests (and get $\frac{n (n-1)}{2}$ p-values) and use one of the p-value correction algorithms to adjust them.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.