15
$\begingroup$

Suppose we split a dataset into 3 parts (train, validation, and test). I know that it's important to make sure the test set doesn't influence our decisions during model selection or hyperparameter tuning or else we may end up overfitting the test set and have unrealistic results. So it's clearly wrong if we tested some model then try to change its hyperparameters and train, validate, and test it again on the same test set.

However, what if we trained and validated, for example, 5 different models, and we decided that we won't modify any of them again. Then we tested each of the 5 models on the (same) unseen test set. If we select the model that achieves the best test result, isn't this the same as if we try different hyperparameter combinations and select the one that has the best performance on the test set?

In this sense, is it wrong when research papers propose multiple methods, test them on the same data, and decide that one of them is the best because it has the best performance on the test set? Isn't it supposed that the test set doesn't influence the choice of the best method?

But if this is wrong, how are we supposed to compare methods (from different papers) on the same test set without being biased? I feel there is a contradiction regarding this point.

$\endgroup$
5
  • 1
    $\begingroup$ Have you found an answer to this? I am having a similar question. Essentially this is what happens to any data set we use right? E.g. Anyone can download the MNIST data and create a train, validation and test split. Say we created a model (e.g. linear regression) and it performs poorly on our test MNIST. We learn from this, and create a deep learning model and it performs better now on our test set. But, we cannot report this as our generalization error because in some way we've overfitted on the test set. $\endgroup$
    – woowz
    Aug 30, 2021 at 6:03
  • 4
    $\begingroup$ You can see this also in the state of the art benchmarks too e.g. the SQUAD (Stanford Question and answering Data set). Essentially, each time a new model is released we are overfitting on the test set. So we can't report that as our true generalization error. Essentially the test set becomes more like a validation set. Do you agree? $\endgroup$
    – woowz
    Aug 30, 2021 at 6:09
  • 1
    $\begingroup$ Unfortunately I haven't found an answer yet. I strongly agree with your opinion regarding the state of the art. I believe the best way to really test the model is to deploy it in a real production environment and see how it works with real data. However, for papers that have not deployed their models, I think the results will always be at the risk of overfitting. $\endgroup$ Aug 31, 2021 at 15:19
  • 1
    $\begingroup$ I vote to reopen this question since changing a model and selecting a model are not equivalent. I still find the answers to the question helpful and relevant here as well - the questions yet are not identical/duplicate. $\endgroup$ Jan 31, 2022 at 19:34
  • $\begingroup$ I agree with @NikolasRieble. $\endgroup$ Feb 4, 2022 at 17:55

2 Answers 2

12
$\begingroup$

You could consider the model itself a hyperparameter as well. If you optimize the hyperparameter using the test set, and then choose the best model, you overfit with the human in the loop.

I like the sklearn documentation on model selection which sports the following chart:

model selection

And futher states:

When evaluating different settings (“hyperparameters”) for estimators, such as the C setting that must be manually set for an SVM, there is still a risk of overfitting on the test set because the parameters can be tweaked until the estimator performs optimally. This way, knowledge about the test set can “leak” into the model and evaluation metrics no longer report on generalization performance. To solve this problem, yet another part of the dataset can be held out as a so-called “validation set”: training proceeds on the training set, after which evaluation is done on the validation set, and when the experiment seems to be successful, final evaluation can be done on the test set.

Additionally, I recommend you to read:

Note that while you call your holdout set test, that is what others call evaluation set. In this context, have a look at MFML 069 - Model validation done right

$\endgroup$
2
  • 1
    $\begingroup$ Fantastic response. The statement that evaluating the performance of multiple models being used for model selection represents leakage of the test set is especially insightful. $\endgroup$ Jan 31, 2022 at 13:02
  • 2
    $\begingroup$ +1. I think it is very important to stress your point on "consider the model itself a hyperparameter". In that sense, to go back to the OP's point, only the "best" model is reported on the test set. Using the test set to select the best model and then report that performance, is using the test set to select "a hyperparameter". $\endgroup$
    – usεr11852
    Oct 20, 2022 at 15:21
2
$\begingroup$

You are correct that the measurement of generalization error should happen after a final model has been selected, on data that has never been passed to any models yet.

This is well explained in Aurelion Geron's book, Hands-on ML with Scikit-Learn, Keras, and Tensorflow. Here's a link to an accompanying Jupyter notebook.

Excerpt (emphasis mine):

Once you are confident about your final model, measure its performance on the test set to estimate the generalization error. Don't tweak your model after measuring the generalization error: you would just start overfitting the test set.

Here, we can consider swapping Model 1 for Model 2 based on the test-set results as a heavy-handed way of "tweaking" the model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.