Evaluating the classifier on K validation sets, but training it on a fixed training set, when data is imbalanced I'm training a binary classifier on imbalanced data (The real/production data has ~%2 of positive labels).
Besides the questionable efficiency of oversampling/undersampling technique, I have a lot of training data, so I can manually add real positive observations to the data, instead of synthesizing them with an oversampling technique.
My assumptions, generally based on intuition, are:

*

*The model should be trained on a dataset with more than 2% of positive labels


*The test set should be as similar as possible to the real data (in this case, to have the same
proportion of positive labels (~2%))


*The validation set should be as similar as possible to the test set.
When balancing the training set by manually adding real positive examples (~35% positive label) and then applying  CV to this data, I violated my second assumption because the positive label proportion in the validation folds was much bigger than the test set.
Another approach I have tried was splitting the dataset into one train set, one validation set, and one test set (hold-out validation), so all my assumptions were kept. However, in this approach, the validation set (and test set) have few observations of the positive labels (less than 50 observations ), and my concern is that kind of overfitting will occur (the model will know to recognize and classify just these few observations as positive labels and having trouble classify new positive observations).
This process made me think about the following approach: create a fixed training set that includes more proportion of positive labels than the real data, and evaluate the model on  K validation sets that have the actual minority label ratio. Similar to CV but with a big difference: the model will be trained on the same training fold each time.
Does my approach may work, or is there a based-on-literature approach to handling this situation?
 A: I think your approach defeats the purpose of cross-validation which is that you can use the same samples for training and validation (in different folds) to get the best possible model out of your limited data. If you have a lot of data, it makes more sense to just have one large validation set that is seperate from the training data instead of K small ones.
A: Class imbalance causes tremendous confusion when it really shouldn’t. Briefly, the likely explanation is that you should not do anything about the class imbalance. Do your modeling of the imbalanced data while evaluating your probability predictions using proper scoring-rules. As is discussed within material cited within these links, a major driver of class imbalance seeming like a problem comes from using discontinuous, improper scoring rules like accuracy.
Once you have good probability predictions, you might choose to set a threshold (perhaps not the software-default of $0.5$) to make hard classifications, depending on whether the predicted probability is above or below the threshold. However, when you do real machine learning, there is a cost associated with incorrect classifications, and the decisions that optimize cost might even involve multiple thresholds, despite the original problem being binary.
For the record, I say that even accuracy can be wrestled with to be perfectly descriptive, even in the presence of considerable imbalance, yet accuracy is problematic, even with perfectly balanced classes.
(That Stephan Kolassa in a few of the linked answers has written extensively about this topic, and he is one of the people from whom I learned about this topic.)
Splitting your out-of-sample data into multiple groups could make sense if your goal is to get an idea for the variability of your performance. A competing approach could be to bootstrap that one out-of-sample set and apply your trained model on each of those bootstrap samples. Alternative approaches using bootstrap exist, too, at the expense of computing time, and there are debates about this beyond computing time. However, I think you’ve made a mistake in handling the imbalance before you reach that point.
