Cross validation and imbalanced learning

I was provided with a heavily imbalanced medical dataset (90-10% proportion among the negative/positive classes) to perform classification.

In order to mitigate the imbalance, I have resorted to oversampling the minority class through SMOTE in order to obtain a balanced dataset. Since I needed cross-validated results, I performed oversampling only on the training partition, leavining the test partition untouched.

The problem is that since the proportion of positive/negative examples changes from train to test, the classifier behaves poorly because it somehow learns the frequency of the two classes in training, and then wrongly uses this notion in the test phase (producing a lot of false positives).

Any idea how I can overcome this problem?

Say the dataset is composed of $N$ and $P$, negative and positive, respectively, with $|P| = \frac{1}{9} |N|$ in the dataset, but with the true-life ratio being $|P| = \alpha |N|$ for some $\alpha > \frac{1}{9}$ (e.g., $\alpha=1$ means that, in real life, positives and negatives are approximately as frequent).

Partition the negative samples into two parts, $N_1$, $N_2$, s.t. $|N_2| = \alpha |P|$.

For example, in the following figure, $N_1, N_2, P$ are the parts in blue, cyan, grey, respectively.

To perform 3-fold CV, for example, partition each of the parts into 3. The first fold, for example, would consist of the top 2 blue and top 2 gray for train, and the bottom 1 cyan and bottom 1 gray for test.

Note that the test set is $\alpha$ balanced. In the train set, you'd use SMOTE to $\alpha$ balance as you're doing now.

(Of course, you can adapt this to other methods besides k-fold.)

Unfortunately, for the percentages you mention, the test will probably be relatively noisy (unless your dataset is large). Personally, I don't see a way around that.

• Should alpha be estimated through cross-validation as well? Anyway thanks, this seems a good solution to better estimate out-of-sample performance since the cyan part doesn't get involved in training the model. Am I right?
– mp85
Oct 19, 2016 at 14:57
• @mp85 I don't see how it's possible to set $\alpha$ through CV - $\alpha$, IIUC, should match what you think will happen in the real world when you deploy your algorithm. As for the cyan part - yes, you're right. Oct 19, 2016 at 15:09
• Alright, now everything's crystal clear. Thank you. I will try to implement your suggestion and see what happens.
– mp85
Oct 19, 2016 at 15:20
• @mp85 Good luck. I'm curious how it will work out, so, if you wan, feel free to update. I think the noisiness of your test dataset (for the parameters you wrote) is an inherent problem with your problem in any case, so am not sure what will work. Oct 20, 2016 at 7:17
• Actually it's not that trivial because I'm using Python's scikit-learn and they have their own routines for cv, so I should implemented my own ones. If I succeed I will update you for sure.
– mp85
Oct 20, 2016 at 8:11