I have 2 versions of the same dataset, one which is fully balanced and one in which the positives:negatives is 1:2.

In both cases, when I train my SVM classifier I get low recall and quite high precision, specifically:

  • balanced: 0.7 recall and 0.87 precision
  • imbalanced: 0.52 recall and 0.82 precision

I would have expected to increase the recall in the imbalanced dataset, as there are now many more negatives to be misclassified than positives (see here as well). Could anyone explain this behaviour? Can I explain it as fewer positives means it's harder for the classifier to learn what a positive looks like?

  • $\begingroup$ Are these P:R values evaluated on an independent test? If yes, how large? If not, are these created through resampling? Please note that: 1. Training set performance is (mostly) meaningless. 2. Rebalancing the training data is at times frowned up but rebalancing the test set is outright wrong. So what are we evaluating against? $\endgroup$
    – usεr11852
    May 22, 2020 at 16:26
  • $\begingroup$ This is performance on leave-one-out CV - I do not have a validation set at the moment, so I use nested CV to find the optimal model parameters and then I reapply to the entire set (similar to here: stats.stackexchange.com/questions/11602/…). The total set has 269 observations when balanced and 208 when unbalanced, which is why I did not want to use hold-out. The observations I removed is because I was worried they might be corrupting my model by being "too good" because of their origin. $\endgroup$
    – mm523
    May 22, 2020 at 16:41


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.