# Machine Learning Model Comparison - Doubts applying statistical tests

I have trained a bunch of classifiers with nested cross-validation and now I'd like to perform some statistical tests to see if their difference in performance is meaningful. So I initially stumbled into this paper, which lead me to this, and so on. Looks like a simple paired t-test is not enough. I was thinking about performing McNemar's Test, 2x5CV F-Test possibly, corrected paired t-test definitely.

I'm quite confused about the methodology to conduct the tests, though. So please bear with me and tell me if my reasoning makes any sense.

So say I have 2 different models already trained, a dataset that I used to perform training/model selection/model evaluation with nested cross-validation, and another that I can use to perform statistical testings.

I would:

- use the fresh dataset (I suppose I can use the initial one too?);
- pick some number of repetitions (say 100);
- pick a test (say McNemar);
- random sample from the dataset 100 times (with or without replacement?);
- compute the performance of both models on that sample;
- compute the McNemar statistic from that same sample;
- report statistics of the 100 McNemar trials (mean, variance, CIs and whatnot).


Does this process make any sense to you? Any help or thoughts appreciated, thank you.

UPDATE Maybe I should point out that my classifiers are already trained (i. e. if you read the paper from Dietterich (first link), I'm in situation 3). This means that I cannot use the corrected paired T-test since it's a method to compare learning algorithms, not classifiers. Dietterich's paper states that I should be using McNemar's test to assess if the performance of the classifiers differs. Still, I have trouble understanding not the test itself (which is pretty simple), but the methodology in order to carry it on properly.

Finally, this is only valid for TWO classifiers. If you have more than 2 things are more complex. You will need to perform all pairwise 2-classifiers tests (and get $\frac{n (n-1)}{2}$ p-values) and use one of the p-value correction algorithms to adjust them.