What happen if I choose the hyperparameters of a classifier based on lowest generalization error?

Question

In this question, the OP asked about a situation that he/she combined training and test datasets into an agumented dataset and then tuned the hyperparameters for best accuracy and then use the training dataset only to train the final model and check its accuracy on test dataset. My situation is somewhat similar to this question but the difference is that I DID NOT combine training and test datasets to form a agumented dataset. Furthermore, my training and test datasets are completely independent and more importantly test dataset is not subset of training dataset. In fact, my test dataset is given by another research group for the same phenomenon as my training dataset. In my situation I used only the training dataset to train the model for each set of the hyperparameters and then measure the accuracy on test dataset. My objective is oriented towards searching the hyperparameters space to have the highest accuracy on test dataset in each iteration of my search. So, to be precise, my model during hyperparameters tuning did not use test dataset for training task, but I stopped the searching of hyperparameters space, when I achieved the highest accuracy on test dataset. So, my question is: Am I overfitting because my final objective is to have the best generalization accuracy? or Do I have some sort of data leakage because of my hyperparameters searching objective is defined based on having the highest generalization accuracy? Any suggestion or thought is appreciated. Here, I created a pseudo-code to introduce my purpose more easily:

DEFINE HYPERPARAMETERS RANGE

DO HYPERPARAMETER TUNING:

  FOR EACH HYPERPARAMETERS IN DEFINED RANGE:

     TRAIN THE CLASSIFIER BY USING ONLY THE TRAINING DATASET

     APPLY THE TRAINED MODEL ON TEST DATASET AND MEASURE THE ACCURACY

     STOP THE SEARCHING IF YOU ACHIEVED THE HIGHEST ACCURACY FOR TEST DATASET

Answer 1

Over-fitting is not the problem here, the problem is that the estimate of generalisation performance will be optimistically biased as the test set has been used to tune some aspect of the model (in this case the hyper-parameters). Whether you will have encountered over-fitting in model selection depends on how many hyper-parameters you have to tune, the sizes of the training and test sets and the sensitivity of your hyper-parameter tuning criterion to over-fitting. Note that over-fitting in model selection can result in a model that over-fits the data or under-fits it.

I wrote a paper to clarify a few of these issues:

G. C. Cawley and N. L. C. Talbot, Over-fitting in model selection and subsequent selection bias in performance evaluation, Journal of Machine Learning Research, 2010. Research, vol. 11, pp. 2079-2107, July 2010. (www).

Note that there often isn't really a statistical distinction between parameters and hyper-parameters (just one set of parameters for which there is a computationally efficient procedure for finding the optimal values, given the values of a second set, which we call hyper-parameters - see my answer on a related question here). So the problems with tuning parameters or hyper-parameters on the test data are actually very similar.

Answer 2

Short answer: yes, you are overfitting.

Your model has a full set of parameters that are typical split into parameters and hyperparameters (but note that this distinction is somewhat arbitrary). You essentially fitted the former to the training data and then conditional on the testing data for you fit the latter to the "test" data.

How much you are overfitting is hard to say (depends in a complicated way e.g. on the size of the "test" data vs. number/influence of hyperparameters), which is why one should not do that and claim/believe/hope that the test set still truly provides an estimate of the generalization error.

What is generally better is to optimize hyperparameters on a validation set (or via cross validation) and then to use a "true" test set to determine how all the final model choice works.

Answer 3

Terminology use around sets in model building is inconsistent, so I will refer to Wiki on sets and SE Data Science question where 'test set' is specifically a held out set only assessed at the very end. In this syntax then what we are discussing is the 'validation set', used as an intermediate check on the calibration of the model using the training set (and to tune hyperparameters).

Am I overfitting because my final objective is to have the best generalization accuracy?

Thanks to the ever present issue of bias and noise overfitting is unavoidable, it is generally a question of whether it is with acceptable limits.

Imagine the ideal; an unbiased dataset where the only errors in the data are completely random and independent between samples (or any other hierarchy within your dataset). Even in this scenario you will still get cross talk between samples due to random overlap in noise. For example Gaussian noise between independent measurements reduces by the square root of the number of measurements taken. This means that any chance coincidences between the two sets will be selected for using the training set even though the validation set was not used to get coefficients for a model. Now, if random noise is low (or sample numbers quadrupled to halve the noise, since noise cancels by square root) in this scenario the overfitting would be minimal, but this is a ideal dataset far removed from any real world.

If your validation set is simply a subset of your full dataset cordoned off from the training phase then the validation set will share the same biases as the training set. This means that the test set will further ingrain these in the model, but then again it would never be able to identify these common biases if used once. In such as case your cannot predict any generalization capability, because you are not generalising your model on independent data.

Do I have some sort of data leakage because of my hyperparameters searching objective is defined based on having the highest generalization accuracy?

Yes, biases will leak strongly, especially where the validation set is not completely independent of the training set. In such a case biases will not be controlled for and will give a falsely optimistic performance estimation. Independent random noise will leak but can be minimised by using large sample numbers.

There is no such thing as a blank-cheque generalisation licence for models, you do not prove boundless generalisability. Strictly speaking a model can only be considered validated for generalised use in a situation/population for which an independent data collection has been carried out. Furthermore, the utility is only validated for the limits of the independent set. E.g. a model built using East Asian data will not be able to demonstrate generalisation outside of East Asia. Before knowing if it is relevant to North America it needs validation on North American data. You can replace East Asia/North America with any combo relevant to your model (Male/Female, working age/retired, poor/rich cats/dogs etc). This is because different limits may have different underlying data generating processes (control variables, covariance structures, biases etc) that fall outside the calibrated range, i.e. you are extrapolating the model beyond its expected limits.