Can I compare different hyperparameters combination performance on testing dataset

Question

I am runing a GridSearch with different range of hyperparameter values in order to find the best ones based on performance metrics (F1, AUC, etc.). I however have an imbalanced dataset, so I need to undersample my training dataset to have a balanced amount of binary (0 or 1) event and non-event samples for the training.

For this reason, I am wondering if it is OK to train multiple models on the training dataset (with different hyperparameter values) and compare their performance on the testing dataset to chose the best hyperparameter values.

I know the "normal" method is to chose hyperparameters on the validation dataset (with cross-validation on a portion of the training dataset), but I feel like it would make more sense to chose them based on their performance on the testing dataset, considering it is not balanced (just like the future real world data that will be used as inputs in the model).

According to my reflexion, it would be OK because I would still train my model on the training dataset and test it on unseen data (testing dataset).But I would like to confirm that it does make sense?

Answer 1

The important consideration is that if you use a set of data to decide something (the algorithm to use, the hyperparameters to use, the preprocessing to use) you cannot trust the metric used to select the choice as a good estimation of the future performance of your classifier (using those choices you made by testing on that set of data). So you can use the test set to choose, but you cannot/should report that AUC or F1 as the expected AUC or F1 on future data.

Assume that your data is purely random with 50% chance for each class (in the limit, with infinite data). You split a finite dataset in training and test sets, and you test different hiperparameters for a particular algorithm (or different algorithms or different preprocessing).

Then each run will have some accuracy (or whatever other metric) around 50% but not exactly 50%. How not exactly depends on how few data you have on both training and test sets, and details of the classification algorithm. But some runs will be above and some will be below 50%. As any rational experimenter, you will choose the run (the set of hyperparameters) that resulted in the highest accuracy. And that is a reasonable choice, and we all do that. But you can see how you should not trust the high accuracy you measured in that run -by construction that value was some random noise/perturbation added to the 50% accuracy you should expect.

The same is true even in examples less artificial than this one, you can and should select the alternative with better metrics, but once you use a dataset to make a choice, you should not trust that value of the metric as expected future values.

Answer 2

The short answer is no; if you choose an option (hyperparameters) based on results for a particular data set then that data set should be regarded as the test set. Any decisions made should be regarded as part of the training process and so need to be verified on a further data set.

An exception might be where the data set is believed to be large enough or similar enough to the target data. Whether you can justify that depends on the application and level of security.

Another exception might be where all your models performed acceptably well on the test data set, according to some acceptance criteria. In that case, although you are choosing one for deployment, you have confidence that the decision doesn't affect the acceptability of the solution.