I am working on classification for highly imbalanced data. Let's say I have a strategy to oversample/undersample the training data. I plan to use an SVM classifier to perform the classification.
Now, since I oversampled/undersampled my training data, it's distribution is now different from the true distribution.
For selecting the hyperparameters for SVM, I understand that the validation data needs to be distributed according to the true distribution, as does the test data.
In the regular situation, if the training data is distributed according to the true distribution, the training and validation data can be combined to perform cross-validation in selecting the hyperparameters.
Now in the situation I am considering, it does not make sense to combine the oversampled/undersampled training data and validation data as they are from different distributions.
So I imagine that a standard way to proceed is simply use the validation data (without combining it to the training data that is oversampled/undersampled) to select my hyerparameters.
I am wondering: are there alternative approaches to select hyperparameters if your training data, in general, is not distributed according to the true distribution?
Some insights appreciated. Thanks!