I am runing a GridSearch with different range of hyperparameter values in order to find the best ones based on performance metrics (F1, AUC, etc.). I however have an imbalanced dataset, so I need to undersample my training dataset to have a balanced amount of binary (0 or 1) event and non-event samples for the training.
For this reason, I am wondering if it is OK to train multiple models on the training dataset (with different hyperparameter values) and compare their performance on the testing dataset to chose the best hyperparameter values.
I know the "normal" method is to chose hyperparameters on the validation dataset (with cross-validation on a portion of the training dataset), but I feel like it would make more sense to chose them based on their performance on the testing dataset, considering it is not balanced (just like the future real world data that will be used as inputs in the model).
According to my reflexion, it would be OK because I would still train my model on the training dataset and test it on unseen data (testing dataset).But I would like to confirm that it does make sense?