0
$\begingroup$

I have a dataset and I'm working on a binary classification task with it. It has a class imbalance problem where False class versus True class is 10: 1.

If I train a neural network on it directly without tackling the imbalance problem, I got the following result:

test LogLoss 0.2025
test AUC 0.8578
              precision    recall  f1-score   support

       False       0.94      0.99      0.96   2294923
        True       0.84      0.30      0.44    224278

    accuracy                           0.93   2519201
   macro avg       0.89      0.65      0.70   2519201
weighted avg       0.93      0.93      0.92   2519201

After I added class weight to the classifier training process, i.e., class_weight={0: 1., 1: 10.}, I retrained the model and got the following result:

test LogLoss 0.4166
test AUC 0.8646
              precision    recall  f1-score   support

       False       0.97      0.81      0.88   2294923
        True       0.28      0.74      0.41    224278

    accuracy                           0.81   2519201
   macro avg       0.62      0.78      0.65   2519201
weighted avg       0.91      0.81      0.84   2519201

It seems the log loss is worse but the AUC is better. The True class's precision is worse but recall is better.

  • How do you explain these changes in metrics, why some are better and some worse?
  • Based on the result,should I use class weight in the training?
$\endgroup$
  • 3
    $\begingroup$ they change differently because they are different measures. If they would behave the same, there would not be a need for so many of them. Which is the better model depends on your problem, how much do you care about false positive/negative, calibration etc. That's a 'business' decision, not a machine learning decision. If you want to understand your model then plot distributions of unthresholded predictions for each group. You can also do it before and after adding weights to see how the prediction change $\endgroup$ – rep_ho Oct 23 '19 at 10:01
  • $\begingroup$ For the "True" instances (the minor class), after using class weight, its precision decreases from 0.84 to 0.28. I found this strange. The class weight should help the classifier better tell the differences between the two classes. Shouldn't all the precision metric increase? $\endgroup$ – Tyler傲来国主 Oct 23 '19 at 10:40
  • 1
    $\begingroup$ that's caused by where you put your threshold. Since you include weights, now you are predicting more subjects as positive thus precision goes down and recall goes up. If you just plot two histograms for predictions of positive and negative classes and look where your threshold (0 or 0.5) is with respect to these distributions, everything will be much clearer to you $\endgroup$ – rep_ho Oct 25 '19 at 11:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.