0
$\begingroup$

I am working with multiple strongly imbalanced binary data sets (# of majority class > 20x # of minority class). Although all the data sets are strongly imbalanced, the ratio of the classes differs between data sets (Ex. One data set has 360/5600 examples as minority, another has 120 out of 6400 minority).

Can you can compare model performance based on F1 scores across strongly imbalanced data sets with slightly different class ratios? I am asking, for example, if I got an F1 score of 0.3 on data set A with class ratio 360/5000 and an F1 score of 0.6 on data set B with class ratio 120/6400, can I say that the classification model performed better on data set B?

If not, is there another performance metric that can be used as such?

$\endgroup$
1
$\begingroup$

F1 is a suitable measure of models tested with imbalance datasets. But I think F1 is mostly a measure for models, rather than datasets. You could not say that dataset A is better than dataset B. There is no better or worse here; dataset is dataset.

$\endgroup$
0
$\begingroup$

F1 score is used in the case where we have skewed classes i.e one type of class examples more than the other type class examples. Mainly we consider a case where we have more negative examples that positive examples. We calculate the F1 value by changing the threshold classifier value. The more the F1 values, the better it performs. In your case, data set B has higher F1 values. So, it will have better performance than A.

$\endgroup$
0
$\begingroup$

In my opinion, it always depends on the application when we say model A is better than model B.

I mean it’s biased to evaluate models according to a single metric, like F1 though it’s the combination of precision and recall. For example, in cases where you may want as many as instances of minority class been correctly classified, then you may use recall as the metric. In some other cases, you may want a higher precision among instances been classified as the minority class, then you can use precision.

Particularly, for class-imbalance data, AUC is a commonly used metric which takes care of the rank of positive class instances and negative ones. Take the minority class as the positive class, you may want to give higher confidence score to positive instances than negative ones. I personally think AUC is a more preferable metric than F1 in your context.

Hope it helps.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.