Keras: How do I train a binary classification net where one class is much more frequent then the other?

Question

I want to train a binary classification net (for NLP) where one class is much more frequent then the other (using Keras).

I have learned that in this case (one class is much more frequent then the other) it is not useful to talk about accuracy. It is more useful to talk about F1 score or to look at ROC curves or talk about the ROC AUC value.

The problem is that you can not directly optimize for those values (loss function can not be given). But what do I have to do in this case? What is the workflow to get a good F1 Score or high ROC AUC value?

Should I use class_weight argument of the fit function in Keras in this case?

Answer 1

You have two ways of approaching this problem:

1. You can use the class_weight parameter. If you input the correct weights this will cause each example from the minority class to impact the loss more than one from the majority class. This in turn will cause the model to be trained well on both classes. To figure out the appropriate weights you can use scikit-learn's compute_class_weight function.
2. You can re-sample the data. The most typical form is over-sampling the minority class (i.e. increase the number of training examples in the minority class), but you can also under-sample the majority class (i.e. reduce the number of training examples in the majority class). imbalanced-learn is a python library that might help you with this.