To check if the performance difference between two method on a single dataset is statistically significant, one common approach in literature seems to be applying cross-validation (or repeated CV), obtain say 25 measurements for each method, then compare these two sets of 25 measurements using wilcoxon signed-rank or t-test etc.
However in one paper I recently came across, they had to use same testset as another study. So for comparison, for each method, they trained multiple models on training set, each of which independently divided the training set to create a validation set to tune hyper-parameters. Then all of these models' accuracy on fixed test set was used as measurements for that method for method comparison.
My question basically, Is this ok? and is this common in literature?
Some of my thoughts on this are:
- Results will be pessimistic, since not all of training set is used. But since it will be equally pessimistic for each method, for model comparison it shouldn't matter.
- Those 25 measurements used in statistical tests will be even less independent than what cross-validation would've produced.
EDIT: This is a "medium"-sized dataset,total no of sample is 8000, testset contains 1600 samples. And, my concern more specifically is whether statistical tests run on these datasets is reliable, or contain more information than a single point estimate of accuracy. Reference to existing literature in particular would be really helpful.