Is it wrong to compare multiple models on the same test set and choose the best model?

Question

Suppose we split a dataset into 3 parts (train, validation, and test). I know that it's important to make sure the test set doesn't influence our decisions during model selection or hyperparameter tuning or else we may end up overfitting the test set and have unrealistic results. So it's clearly wrong if we tested some model then try to change its hyperparameters and train, validate, and test it again on the same test set.

However, what if we trained and validated, for example, 5 different models, and we decided that we won't modify any of them again. Then we tested each of the 5 models on the (same) unseen test set. If we select the model that achieves the best test result, isn't this the same as if we try different hyperparameter combinations and select the one that has the best performance on the test set?

In this sense, is it wrong when research papers propose multiple methods, test them on the same data, and decide that one of them is the best because it has the best performance on the test set? Isn't it supposed that the test set doesn't influence the choice of the best method?

But if this is wrong, how are we supposed to compare methods (from different papers) on the same test set without being biased? I feel there is a contradiction regarding this point.

Answer 1

You could consider the model itself a hyperparameter as well. If you optimize the hyperparameter using the test set, and then choose the best model, you overfit with the human in the loop.

I like the sklearn documentation on model selection which sports the following chart:

And futher states:

When evaluating different settings (“hyperparameters”) for estimators, such as the C setting that must be manually set for an SVM, there is still a risk of overfitting on the test set because the parameters can be tweaked until the estimator performs optimally. This way, knowledge about the test set can “leak” into the model and evaluation metrics no longer report on generalization performance. To solve this problem, yet another part of the dataset can be held out as a so-called “validation set”: training proceeds on the training set, after which evaluation is done on the validation set, and when the experiment seems to be successful, final evaluation can be done on the test set.

Additionally, I recommend you to read:

• How is cross validation different from data snooping?
• Bonferroni Correction & machine learning

Note that while you call your holdout set test, that is what others call evaluation set. In this context, have a look at MFML 069 - Model validation done right

Answer 2

You are correct that the measurement of generalization error should happen after a final model has been selected, on data that has never been passed to any models yet.

This is well explained in Aurelion Geron's book, Hands-on ML with Scikit-Learn, Keras, and Tensorflow. Here's a link to an accompanying Jupyter notebook.

Excerpt (emphasis mine):

Once you are confident about your final model, measure its performance on the test set to estimate the generalization error. Don't tweak your model after measuring the generalization error: you would just start overfitting the test set.

Here, we can consider swapping Model 1 for Model 2 based on the test-set results as a heavy-handed way of "tweaking" the model.