1
$\begingroup$

I am runing a GridSearch with different range of hyperparameter values in order to find the best ones based on performance metrics (F1, AUC, etc.). I however have an imbalanced dataset, so I need to undersample my training dataset to have a balanced amount of binary (0 or 1) event and non-event samples for the training.

For this reason, I am wondering if it is OK to train multiple models on the training dataset (with different hyperparameter values) and compare their performance on the testing dataset to chose the best hyperparameter values.

I know the "normal" method is to chose hyperparameters on the validation dataset (with cross-validation on a portion of the training dataset), but I feel like it would make more sense to chose them based on their performance on the testing dataset, considering it is not balanced (just like the future real world data that will be used as inputs in the model).

According to my reflexion, it would be OK because I would still train my model on the training dataset and test it on unseen data (testing dataset).But I would like to confirm that it does make sense?

$\endgroup$
3
  • $\begingroup$ You do not need to balance an imbalanced dataset, especially if AUC is a metric of interest. $\endgroup$ Dec 13, 2021 at 14:27
  • $\begingroup$ @DikranMarsupial, could you please elaborate what you mean? I know the testing dataset doesn't need to be balanced, but are you saying that I don't need to balance my training dataset neither even though I have much more non-event (0) than event (1)? $\endgroup$
    – Boocaj
    Dec 13, 2021 at 16:13
  • $\begingroup$ Yes, for most of the classifiers I have tried it makes very little difference. Class imbalance only tends to be a real problem if the dataset is very small. $\endgroup$ Dec 13, 2021 at 16:17

2 Answers 2

1
$\begingroup$

The important consideration is that if you use a set of data to decide something (the algorithm to use, the hyperparameters to use, the preprocessing to use) you cannot trust the metric used to select the choice as a good estimation of the future performance of your classifier (using those choices you made by testing on that set of data). So you can use the test set to choose, but you cannot/should report that AUC or F1 as the expected AUC or F1 on future data.

Assume that your data is purely random with 50% chance for each class (in the limit, with infinite data). You split a finite dataset in training and test sets, and you test different hiperparameters for a particular algorithm (or different algorithms or different preprocessing).

Then each run will have some accuracy (or whatever other metric) around 50% but not exactly 50%. How not exactly depends on how few data you have on both training and test sets, and details of the classification algorithm. But some runs will be above and some will be below 50%. As any rational experimenter, you will choose the run (the set of hyperparameters) that resulted in the highest accuracy. And that is a reasonable choice, and we all do that. But you can see how you should not trust the high accuracy you measured in that run -by construction that value was some random noise/perturbation added to the 50% accuracy you should expect.

The same is true even in examples less artificial than this one, you can and should select the alternative with better metrics, but once you use a dataset to make a choice, you should not trust that value of the metric as expected future values.

$\endgroup$
1
  • $\begingroup$ Thank you for your answer. I understand what you are saying. Would it however be acceptable to use the testing dataset to make the hyperparameter value choices and than re-run the predicting model (with the selected hyperparameter values) multiple times (lets say 20 iterations) with those selected hyperparameters and than establish the performance of the model by averaging the 20 iterations performance on a selected metric (F1 score for example)? $\endgroup$
    – Boocaj
    Dec 13, 2021 at 13:36
0
$\begingroup$

The short answer is no; if you choose an option (hyperparameters) based on results for a particular data set then that data set should be regarded as the test set. Any decisions made should be regarded as part of the training process and so need to be verified on a further data set.

An exception might be where the data set is believed to be large enough or similar enough to the target data. Whether you can justify that depends on the application and level of security.

Another exception might be where all your models performed acceptably well on the test data set, according to some acceptance criteria. In that case, although you are choosing one for deployment, you have confidence that the decision doesn't affect the acceptability of the solution.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.