I am trying to fit my data to a classifier using SVM. My data has 2 classes, the positive class which occurs with a probability of 0.002 and the negative class which is the dominant one. Suppose that I can generate as much samples as I need from both classes to train the classifier well.
I decided to try oversampling the positive class so that the ratio between the two classes is roughly 1:1. Since I'm new to this, I initially used a test set with the same ratio for both classes. Since my original data set is imbalanced, I am using the precision as a measure of performance. For 25000 data points for both classes, my validation test reported a 96% precision (for the positive class which is the rare one) which I'm happy about. In this scenario, my validation data also had a roughly 1:1 ratio between the 2 classes.
Now I learned, from reading the posts here, that oversampling should only be done on the training data. So I then generated a test data that faithfully represents the distribution. Out of 25000 validation samples, only around 50 are in the positive class. However, the precision I got from validation dropped significantly to 6%.
Does anyone have an idea why there is such a huge discrepancy in the validation results: I was expecting that using validation data where there is a 1-1 ratio in both classes was really going to test the performance of the classifier in the positive class.
Any thoughts on why this may occur? It will help me overcome with imbalanced data.
Thanks for any suggestions!