You have a training and a test set. You combine them and do something like GridSearch to decide the hyperparameters of the model. Then, you fit a model on the training set using these hyperparameters, and you use the test set to evaluate it.

Is this overfitting ? Ultimately, the data was not fitted on the test set, but the test set was considered when deciding hyperparameters.

  • 5
    $\begingroup$ This is a case of Data Leakage. Similarly in the statistics the term data snooping (data dredging) - is used [en.wikipedia.org/wiki/Data_dredging]. You leak some information from test set to the train set. This will falsely improve the performance of your model on the test set. $\endgroup$
    – Hooman
    Commented Sep 14, 2018 at 8:25
  • $\begingroup$ Related: stats.stackexchange.com/questions/189678/… $\endgroup$
    – Adrian
    Commented Jul 14, 2020 at 4:36
  • $\begingroup$ Intuitively it would be like scoring a student's capability on a test where the student already know the solutions. It's not overfitting, it's just that you would have no idea about the performance of the model $\endgroup$ Commented Jan 7, 2021 at 18:28

6 Answers 6


The idea behind holdout and cross validation is to estimate the generalization performance of a learning algorithm--that is, the expected performance on unknown/unseen data drawn from the same distribution as the training data. This can be used to tune hyperparameters or report the final performance. The validity of this estimate depends on the independence of the data used for training and estimating performance. If this independence is violated, the performance estimate will be overoptimistically biased. The most egregious way this can happen is by estimating performance on data that has already been used for training or hyperpameter tuning, but there are many more subtle and insidious ways too.

The procedure you asked about goes wrong in multiple ways. First, the same data is used for both training and hyperpameter tuning. The goal of hyperparameter tuning is to select hyperparameters that will give good generalization performance. Typically, this works by estimating the generalization performance for different choices of hyperparameters (e.g. using a validation set), and then choosing the best. But, as above, this estimate will be overoptimistic if the same data has been used for training. The consequence is that sub-optimal hyperparameters will be chosen. In particular, there will be a bias toward high capacity models that will overfit.

Second, data that has already been used to tune hyperparameters is being re-used to estimate performance. This will give a deceptive estimate, as above. This isn't overfitting itself but it means that, if overfitting is happening (and it probably is, as above), then you won't know it.

The remedy is to use three separate datasets: a training set for training, a validation set for hyperparameter tuning, and a test set for estimating the final performance. Or, use nested cross validation, which will give better estimates, and is necessary if there isn't enough data.

  • $\begingroup$ suggested edit: future data -> unknown/unseen data: distribution can change over time (in my field e.g. drift due to detector aging, ...), so we evaluate future performance separately, from generalization error which measures predictive abilities wrt. just unknown data. $\endgroup$
    – cbeleites
    Commented Sep 16, 2018 at 10:08

I would say you are not necessarily overfitting, because overfitting is a term that is normally used to indicate that your model does not generalise well. E.g. If you would be doing linear regression on something like MNIST images, you are probably still underfitting (it does not generalise enough) when training on both training and test data.

What you are doing, however, is still not a good thing. The test set is normally a part of the data that you want to use to check how good the final, trained model will perform on data it has never seen before. If you use this data to choose hyperparameters, you actually give the model a chance to "see" the test data and to develop a bias towards this test data. Therefore, you actually lose the possibility to find out how good your model would actually be on unseen data (because it has already seen the test data).

It might be possible that you do not really care about how well your model performs, but then you would not need a test set either. Because in most scenarios you do want to have an idea how good a model is, it is best to lock the test data away before you start doing anything with the data. Something as little as using test data during pre-processing, will probably lead to a biased model.

Now you might be asking yourself: "How should I find hyperparameters then?". The easiest way would be to split the available data (assuming that you already safely put away some data for testing) into a training set and a so-called validation set. If you have little data to work with, it probably makes more sense to take a look at cross validation


Yes, you are overfitting. The test set should be used only for testing, not for parameter tuning. Searching for parameters on the test set will learn the rules that are present in the test set, and eventually overfit it.

  • 3
    $\begingroup$ You're not overfitting per se, but indulging into data leakage. See other answers. $\endgroup$
    – Firebug
    Commented Sep 14, 2018 at 12:54

It is not necessarily overfitting, but it also runs an unnecessary risk of overfitting, and you deprive yourself of the possibility to detect overfitting.

  • Overfitting happens when your model is too complex/has too many degrees of freedom for the available training data. This includes degrees of freedom for the hyperparameter space you search. So if your data set is still large enough, you don't overfit (say, you have thousands of cases, just two variates to regress on and a single continuous hyperparameter to tune - that would likely still be OK. On the other hand, if you a handful of cases, hundreds or thousands of variates and a large hyperparameter search space, you run a huge risk of overfitting).

  • But as all your data entered the training phase (during the hyperparameter optimization), you lost the chance to measure generalization error and thus cannot check/show that you do not overfit. Which is as bad as overfitting, unless you can give other evidence that you are not in a situation where overfitting can occur.

  • Moreover, you traded in your ability to measure generalization error for at most a minute improvement in training: You could (and should) have done the whole training on the training set - that's what it is for. And training includes fixing the hyperparameters.

  • From that point of view, the decision is really whether you need to have an error estimate based on unknown data or not (again based on the overall risk of overfitting - and in machine learning the decision would pretty much always be that unknown data performance is needed), and then either do the whole training on your data, or do the whole training on the training set and test with the test set. (Or possibly on multiple such train/test splits as in cross validation).


It is an "in-sample" forecast since you eventually make the forecast on observations that are already part of your training set. Why not use n-fold cross-validation? By doing that, at each time, you are making "out-of" sample forecast, in which test set and training set are separate.


The idea of CV is to overcome the weaknesses of Train-Test split (loss of information, only a part being used for testing etc.). Hence, CV ensures that all parts of data falls into training and testing folds in the successive iterations. This ensures that we get a balanced picture of whatever we are trying to evaluate (choice of hyperparameter, algorithms etc.).

Given this situation, you should use entire dataset for tuning. If you use only train fold, the tuned hyperparameter would be specific to Train fold; not the whole data. My experience is that the model would exhibit overfitting even after tuning using only Train set.

imho, practitioners are going overboard with the term 'data leak'. In the effort to prevent data leak, they are giving too much importance to train-test split; which is just a chance. Judicious application of CV is the best approach. Not a combination of Train-Test split and CV.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.