I am trying to fit my data to a classifier using SVM. My data has 2 classes, the positive class which occurs with a probability of 0.002 and the negative class which is the dominant one. Suppose that I can generate as much samples as I need from both classes to train the classifier well.

I decided to try oversampling the positive class so that the ratio between the two classes is roughly 1:1. Since I'm new to this, I initially used a test set with the same ratio for both classes. Since my original data set is imbalanced, I am using the precision as a measure of performance. For 25000 data points for both classes, my validation test reported a 96% precision (for the positive class which is the rare one) which I'm happy about. In this scenario, my validation data also had a roughly 1:1 ratio between the 2 classes.

Now I learned, from reading the posts here, that oversampling should only be done on the training data. So I then generated a test data that faithfully represents the distribution. Out of 25000 validation samples, only around 50 are in the positive class. However, the precision I got from validation dropped significantly to 6%.

Does anyone have an idea why there is such a huge discrepancy in the validation results: I was expecting that using validation data where there is a 1-1 ratio in both classes was really going to test the performance of the classifier in the positive class.

Any thoughts on why this may occur? It will help me overcome with imbalanced data.

Thanks for any suggestions!

  • $\begingroup$ You may have overfit, upsampling a minority class to 1:1 where the minority class occurs only at 0.002 frequency is insane. $\endgroup$ – user2974951 Dec 11 '18 at 8:26
  • $\begingroup$ What recommendations do you have for the ratio between the sample sizes of the 2 classes then? I've tried 10000 samples of the positive class and 100000 samples of the negative class but the best I can get is 76% precision. I'm aiming for high 90's $\endgroup$ – user1237300 Dec 11 '18 at 15:16
  • $\begingroup$ Because of the high imbalance ratio, balancing the data set here is pointless, you will either majorly upsample your minority class or majorly downsample your majority class, either way you will majorly influence your results. Two options as I see it: 1) use an algorithm that can take care of this imbalance internally while training or 2) use a different approach, such as anomaly detection. $\endgroup$ – user2974951 Dec 12 '18 at 10:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.