0
$\begingroup$

This question already has an answer here:

I’m currently encountering some problems analyzing a dataset with neural network. The problem is that I have an unbalanced binary class training set (10:1). Training accuracy for both classes are 100%. When predicting I get accuracy of 99% for the majority class in the validation set, and accuracy of 0.2% for the minority class. So I was wondering if over-sampling would help to improve the accuracy for the minority class.

$\endgroup$

marked as duplicate by jbowman, kjetil b halvorsen, mkt, Peter Flom Oct 19 '18 at 11:00

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 1
    $\begingroup$ I would advice yo against the use of oversampling unless you are will to back-transform the estimated probabilities to account for the oversampling step. Accuracy is a bad metric to focus on, especially in unbalanced sample. Ideally we would to use cost-sensitive approach and/or try using a proper scoring rule like Brier score or Continuous rank probability score (CRPS). $\endgroup$ – usεr11852 Oct 18 '18 at 22:03
1
$\begingroup$

There are a lot of things that would improve the performance on the minority class. The reason you "have a problem", is that your neural network has "realized" that the majority class is much more common. I.e. if the information you feed in reality tells it nothing about the class, it should predict the majority class to do well in terms of accuracy and the provided loss function (presumably cross-entropy). Even when the data can help you distinguish (to some extent) the two classes, the neural network predicts the majority class more often to the extent that it makes sense with respect to the loss function you optimized.

If you don't like what happened you can change the data e.g. by oversampling or undersampling one class (in various ways, e.g. simple oversampling, SMOTE etc.). However, you need to realize that you would likely want to correct for that in the predictions, if you care about well-calibrated predicted probabilities. You could also change your loss function to induce your neural network to realize that you care more than your original loss function implied about whether it gets the minority class wrong.

$\endgroup$
  • 1
    $\begingroup$ The root of the problem is the choice of an improper discontinuous scoring rule. See here. A proper rule will not be hurt by imbalance. $\endgroup$ – Frank Harrell Oct 19 '18 at 11:44

Not the answer you're looking for? Browse other questions tagged or ask your own question.