I have trained a bunch of classifiers with nested cross-validation and now I'd like to perform some statistical tests to see if their difference in performance is meaningful. So I initially stumbled into this paper, which lead me to this, and so on. Looks like a simple paired t-test is not enough. I was thinking about performing McNemar's Test, 2x5CV F-Test possibly, corrected paired t-test definitely.
I'm quite confused about the methodology to conduct the tests, though. So please bear with me and tell me if my reasoning makes any sense.
So say I have 2 different models already trained, a dataset that I used to perform training/model selection/model evaluation with nested cross-validation, and another that I can use to perform statistical testings.
- use the fresh dataset (I suppose I can use the initial one too?);
- pick some number of repetitions (say 100);
- pick a test (say McNemar);
- random sample from the dataset 100 times (with or without replacement?);
- compute the performance of both models on that sample;
- compute the McNemar statistic from that same sample;
- report statistics of the 100 McNemar trials (mean, variance, CIs and whatnot).
Does this process make any sense to you? Any help or thoughts appreciated, thank you.
UPDATE Maybe I should point out that my classifiers are already trained (i. e. if you read the paper from Dietterich (first link), I'm in situation 3). This means that I cannot use the corrected paired T-test since it's a method to compare learning algorithms, not classifiers. Dietterich's paper states that I should be using McNemar's test to assess if the performance of the classifiers differs. Still, I have trouble understanding not the test itself (which is pretty simple), but the methodology in order to carry it on properly.