1
$\begingroup$

I want to train a binary classification net (for NLP) where one class is much more frequent then the other (using Keras).

I have learned that in this case (one class is much more frequent then the other) it is not useful to talk about accuracy. It is more useful to talk about F1 score or to look at ROC curves or talk about the ROC AUC value.

The problem is that you can not directly optimize for those values (loss function can not be given). But what do I have to do in this case? What is the workflow to get a good F1 Score or high ROC AUC value?

Should I use class_weight argument of the fit function in Keras in this case?

$\endgroup$
1
  • $\begingroup$ You can't directly optimize accuracy either, since the function is not continuous. So, how would you optimize the neural network weights when you are interested in maximizing accuracy? Most people would just use cross-entropy, whether measuring performance with accuracy, ROC AUC, or another non-differentiable metric. $\endgroup$
    – Sycorax
    Sep 23 '18 at 22:23
1
$\begingroup$

You have two ways of approaching this problem:

  1. You can use the class_weight parameter. If you input the correct weights this will cause each example from the minority class to impact the loss more than one from the majority class. This in turn will cause the model to be trained well on both classes. To figure out the appropriate weights you can use scikit-learn's compute_class_weight function.
  2. You can re-sample the data. The most typical form is over-sampling the minority class (i.e. increase the number of training examples in the minority class), but you can also under-sample the majority class (i.e. reduce the number of training examples in the majority class). imbalanced-learn is a python library that might help you with this.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.