I was provided with a heavily imbalanced medical dataset (90-10% proportion among the negative/positive classes) to perform classification.
In order to mitigate the imbalance, I have resorted to oversampling the minority class through SMOTE in order to obtain a balanced dataset. Since I needed cross-validated results, I performed oversampling only on the training partition, leavining the test partition untouched.
The problem is that since the proportion of positive/negative examples changes from train to test, the classifier behaves poorly because it somehow learns the frequency of the two classes in training, and then wrongly uses this notion in the test phase (producing a lot of false positives).
Any idea how I can overcome this problem?