ML: Model comparison using statistical tests when testset is fixed

Question

To check if the performance difference between two method on a single dataset is statistically significant, one common approach in literature seems to be applying cross-validation (or repeated CV), obtain say 25 measurements for each method, then compare these two sets of 25 measurements using wilcoxon signed-rank or t-test etc.

However in one paper I recently came across, they had to use same testset as another study. So for comparison, for each method, they trained multiple models on training set, each of which independently divided the training set to create a validation set to tune hyper-parameters. Then all of these models' accuracy on fixed test set was used as measurements for that method for method comparison.

My question basically, Is this ok? and is this common in literature?

Some of my thoughts on this are:

1. Results will be pessimistic, since not all of training set is used. But since it will be equally pessimistic for each method, for model comparison it shouldn't matter.
2. Those 25 measurements used in statistical tests will be even less independent than what cross-validation would've produced.

EDIT: This is a "medium"-sized dataset,total no of sample is 8000, testset contains 1600 samples. And, my concern more specifically is whether statistical tests run on these datasets is reliable, or contain more information than a single point estimate of accuracy. Reference to existing literature in particular would be really helpful.

Answer 1

A single designated test set is very common in the machine learning literature (see e.g. all these leaderboards on paperswithcode.com - I would guess they would [nearly?] all be a single test set). That's also what is done in kaggle competitions.

There are some arguments for and against this. E.g. if you did cross-validation you should ensure that every method uses the same CV scheme (not easy across multiple papers) and that no information from one CV-fold is used in another to drive modeling decision for another fold. That is all a bit more complicated than when you simply have a single test set. You make people's lives easier by "allowing" them to do anything they want to do with the training data to drive just about any modeling decision. In fact, at my company we have occasionally really ring-fenced a test set so that a modeling team can look at data any way they want, but in the end we check performance on data nobody ever looked at.

Additionally, if the training and test set are both huge, then for method comparison the extra precision you get in your comparison from doing CV (or perhaps even better repeated CV) do not matter. Since you trained on the same data and evaluate on the same data, you do not really need to use rank tests on summaries of the results, but can rather directly compare metrics (e.g. accuracy as a binary outcome) - again, if the data are only large enough statistical tests may be less and less relevant (i.e. when differences leading to a rejection of some null hypothesis are so tiny that they are in practice irrelevant - then you can rather only highlight differences that have some real relevance).

For Kaggle competitions it is hard to see what else you could do, if you wish to avoid people knowing the test results (otherwise people can cheat and exploit this knowledge - as has been alleged to have happened in a recent competition, where someone is supposed to have reconstructed the test data).

Problems with having a single test set are probably worst when we talk about small datasets, e.g. you can more easily overfit the test set, the training-test set is rather random and can easily favor one approach over another by chance and so on. Additionally, the evaluation is not very precise/efficient and there is of course a reason why people prefer CV or bootstrapping for hyper-parameter choice, model selection and other similar choices (and the existence of a test set keeps you "honest", because you know that not doing this properly will hurt you on the test set).