You have a training and a test set. You combine them and do something like GridSearch to decide the hyperparameters of the model. Then, you fit a model on the training set using these hyperparameters, and you use the test set to evaluate it.
Is this overfitting ? Ultimately, the data was not fitted on the test set, but the test set was considered when deciding hyperparameters.
The idea behind holdout and cross validation is to estimate the generalization performance of a learning algorithm--that is, the expected performance on unknown/unseen data drawn from the same distribution as the training data. This can be used to tune hyperparameters or report the final performance. The validity of this estimate depends on the independence of the data used for training and estimating performance. If this independence is violated, the performance estimate will be overoptimistically biased. The most egregious way this can happen is by estimating performance on data that has already been used for training or hyperpameter tuning, but there are many more subtle and insidious ways too.
The procedure you asked about goes wrong in multiple ways. First, the same data is used for both training and hyperpameter tuning. The goal of hyperparameter tuning is to select hyperparameters that will give good generalization performance. Typically, this works by estimating the generalization performance for different choices of hyperparameters (e.g. using a validation set), and then choosing the best. But, as above, this estimate will be overoptimistic if the same data has been used for training. The consequence is that sub-optimal hyperparameters will be chosen. In particular, there will be a bias toward high capacity models that will overfit.
Second, data that has already been used to tune hyperparameters is being re-used to estimate performance. This will give a deceptive estimate, as above. This isn't overfitting itself but it means that, if overfitting is happening (and it probably is, as above), then you won't know it.
The remedy is to use three separate datasets: a training set for training, a validation set for hyperparameter tuning, and a test set for estimating the final performance. Or, use nested cross validation, which will give better estimates, and is necessary if there isn't enough data.
I would say you are not necessarily overfitting, because overfitting is a term that is normally used to indicate that your model does not generalise well. E.g. If you would be doing linear regression on something like MNIST images, you are probably still underfitting (it does not generalise enough) when training on both training and test data.
What you are doing, however, is still not a good thing. The test set is normally a part of the data that you want to use to check how good the final, trained model will perform on data it has never seen before. If you use this data to choose hyperparameters, you actually give the model a chance to "see" the test data and to develop a bias towards this test data. Therefore, you actually lose the possibility to find out how good your model would actually be on unseen data (because it has already seen the test data).
It might be possible that you do not really care about how well your model performs, but then you would not need a test set either. Because in most scenarios you do want to have an idea how good a model is, it is best to lock the test data away before you start doing anything with the data. Something as little as using test data during pre-processing, will probably lead to a biased model.
Now you might be asking yourself: "How should I find hyperparameters then?". The easiest way would be to split the available data (assuming that you already safely put away some data for testing) into a training set and a so-called validation set. If you have little data to work with, it probably makes more sense to take a look at cross validation
Yes, you are overfitting. The test set should be used only for testing, not for parameter tuning. Searching for parameters on the test set will learn the rules that are present in the test set, and eventually overfit it.
It is not necessarily overfitting, but it also runs an unnecessary risk of overfitting, and you deprive yourself of the possibility to detect overfitting.
Overfitting happens when your model is too complex/has too many degrees of freedom for the available training data. This includes degrees of freedom for the hyperparameter space you search. So if your data set is still large enough, you don't overfit (say, you have thousands of cases, just two variates to regress on and a single continuous hyperparameter to tune - that would likely still be OK. On the other hand, if you a handful of cases, hundreds or thousands of variates and a large hyperparameter search space, you run a huge risk of overfitting).
But as all your data entered the training phase (during the hyperparameter optimization), you lost the chance to measure generalization error and thus cannot check/show that you do not overfit. Which is as bad as overfitting, unless you can give other evidence that you are not in a situation where overfitting can occur.
Moreover, you traded in your ability to measure generalization error for at most a minute improvement in training: You could (and should) have done the whole training on the training set - that's what it is for. And training includes fixing the hyperparameters.
From that point of view, the decision is really whether you need to have an error estimate based on unknown data or not (again based on the overall risk of overfitting - and in machine learning the decision would pretty much always be that unknown data performance is needed), and then either do the whole training on your data, or do the whole training on the training set and test with the test set. (Or possibly on multiple such train/test splits as in cross validation).
It is an "in-sample" forecast since you eventually make the forecast on observations that are already part of your training set. Why not use n-fold cross-validation? By doing that, at each time, you are making "out-of" sample forecast, in which test set and training set are separate.
The idea of CV is to overcome the weaknesses of Train-Test split (loss of information, only a part being used for testing etc.). Hence, CV ensures that all parts of data falls into training and testing folds in the successive iterations. This ensures that we get a balanced picture of whatever we are trying to evaluate (choice of hyperparameter, algorithms etc.).
Given this situation, you should use entire dataset for tuning. If you use only train fold, the tuned hyperparameter would be specific to Train fold; not the whole data. My experience is that the model would exhibit overfitting even after tuning using only Train set.
imho, practitioners are going overboard with the term 'data leak'. In the effort to prevent data leak, they are giving too much importance to train-test split; which is just a chance. Judicious application of CV is the best approach. Not a combination of Train-Test split and CV.
