I have a imbalance dataset for a classification task, with the minority class accounting for about 21% of the total.

When I use a decision tree based model for prediction, let's say a classification tree or a Random Forest (with scikit-learn), I get a global accuracy of 76%. The detection rate of the minority class (also called sensitivity), is only about 43%.

So, to get an higher detection rate for the minority class, I penalize it more heavily. In sklearn, one can do it with the parameter class_weight={'0':1, '1':penal_value} (here, 0 being the majority class, and 1 the minority).

Of course penal_value is >= 1.

For example, when I set it to 2, I get a 71 global accuracy, but a 55 minority class detection rate.

When I set it to 3.76 (= to 81% / 21%, the rate of the majority on the minority), I get 64 global accuracy, and 60 detection rate.

When I set it to 5, ... Well, you understand the procedure.

What I whant is a measure, that tell what is a "good" value for the penalization. Because it seems that, at the beginning, so with penal_value between 1 and 3.76, it's in some way interesting to lose some global accuracy, because we gain enough sensitivity. It's worth the trade-off.

But past this point, 3.76, it's "not worth" it. The loose in global accuracy "is not compensated" by the gain in sensitivity.

How could I put in a more "objective" way/measure?

  • $\begingroup$ A ROC curve, with points coming from predictions from models with different penalization values could be useful. But it didn't tell us what is the "best" penalization value trade off $\endgroup$ – hellowolrd May 14 '19 at 14:40

When dealing with classification, the most intuitive metric to evaluate a model's perfomance is the accuracy, because it tells how many registers have been classified correctly: the higher, the better. However, as you have noticed, accuracy is not a good measure of the model's performance when dealing with an imbalanced dataset. Imagine that the minority class is the 1% of the total dataset... if your model predicted every register as the majority class, you would get a very high accuracy.

It's preferable in cases such us yours to maximize other metrics rather than accuracy. ROC's AUC (Area Under de ROC Curve) can give you a better idea about how is the model predicting correctly both the positive and the negative class. Moreover, this metric is independent of the classification threshold, in contrast with sensitivity (also known as recall) or especificity. However, if you are specially interested in the performance of the possitive class rather than the negative one, it's recommnended to maximize the F1 Score, the harmonic averagte of precision and recall.

You can read more about all this terms in this article.

Good luck!

  • $\begingroup$ Thanks, would it be a good idea to compute F1 Score for both class, then multiply them and choose the penalization value that maximize it? For example, for different values of weight/penalization, let's say 1 and 3, for minority class, I got the following results : F1_score_minority = 54 and 59. And for F1_score_majority = 89 and 80. So the multiplication is 4806 and 4720. Thus would i be a good idea to select 1 as penalization value? $\endgroup$ – hellowolrd May 16 '19 at 10:02
  • $\begingroup$ No, it wouldn't. In fact, you shouldn't have two F1 scores. As I already said, F1 score is interesting in problems where the positive class is the minority one and you are specially interested in the performance of this one, not the negative, because it only takes into account the positive one, not the negative. Think of F1 score as a combination of recall and precision. Since you only have got one recall and one precision because you've only got one positive class, you should have only one F1 score. If need more help with this, please, show how have you computed the metric $\endgroup$ – alexdefelipe May 19 '19 at 9:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.