3
$\begingroup$

Say I have a binary classification task, where the positive class (1) is only 1% of the whole data set.

Intuitively I can understand why this could be bad for the classifier as the model may learn to just predict everything to be the negative class (0). From my experience and online search results, there seems like to be multiple ways dealing with this (also depends on your model and your evaluation metric. I found gradient boosting e.g. xgboost etc actually did not perform too bad with imbalanced data, but neural net seems like would get affected a lot more).

For example, upsampling the positive class or downsample the negatives, or use class weights which penalize the misclassification on the minority class more. I'm trying to build a neural network model (cnn) on this data set, I did not do any sampling in my approach. I used class weights and performed stratified cross validation so each train set and validation fold contains the same percentage of positive class.

My question is, I'm doing batch training (e.g. with batch size 128), most of the batches would only contain the negative class. How would it affect model learning? Can someone give an intuitive explanation or with equations?

I would guess it's harmful to the model training as in many batches the model only sees negative class, which would lead the model params biased towards class 0 and eventually slows down training if it does not make training much more difficult. If it's the case, I think what I could do is to make sure each batch contains some percentage of positive class by just random sampling.

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ "Intuitively I can understand why this could be bad for the classifier as the model may learn to just predict everything to be the negative class". This, while often repeated, is not correct. Classification models generally learn probabilities, and then you (the human) make class assignments by thresholding those probabilities. If you have a classifier always predicting 0, that's almost certainly because the classification threshold was set carelessly, not because of the balance of the classes in the training data. $\endgroup$
    – Matthew Drury
    Commented Feb 9, 2019 at 4:21
  • $\begingroup$ @MatthewDrury umm, true, I didn't give it too many thoughts, the manual thresholding (not the model with data) could produce totally different stories on metrics like accuracy, recall, precision etc. I guess I didn't really even understand why class imbalance is bad in the first place. $\endgroup$
    – Sam
    Commented Feb 9, 2019 at 4:38
  • $\begingroup$ Me niether: stats.stackexchange.com/questions/247871/… stats.stackexchange.com/questions/285231/… $\endgroup$
    – Matthew Drury
    Commented Feb 9, 2019 at 6:40
  • $\begingroup$ @MatthewDrury well, let's not throw the baby out with the bath water now :-) It's true that (probabilistic) classifiers learn class-conditional probabilities (though there exist non-probabilistic classifiers, which directly learn a classification hyperplane or hypersurface, e.g. SVMs). However, the error on the estimated probabilities is affected by class imbalance (especially the relative error). Think to the extreme case where we have 0 cases of the minority class in our training set. $\endgroup$
    – DeltaIV
    Commented Feb 9, 2019 at 18:46
  • 1
    $\begingroup$ @DeltaIV Do you mean the variance of the estimated probabilities is larger the smaller the population probability is? $\endgroup$
    – Matthew Drury
    Commented Feb 9, 2019 at 20:10

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.