Comparing F1 score across imbalanced data sets

Question

I am working with multiple strongly imbalanced binary data sets (# of majority class > 20x # of minority class). Although all the data sets are strongly imbalanced, the ratio of the classes differs between data sets (Ex. One data set has 360/5600 examples as minority, another has 120 out of 6400 minority).

Can you can compare model performance based on F1 scores across strongly imbalanced data sets with slightly different class ratios? I am asking, for example, if I got an F1 score of 0.3 on data set A with class ratio 360/5000 and an F1 score of 0.6 on data set B with class ratio 120/6400, can I say that the classification model performed better on data set B?

If not, is there another performance metric that can be used as such?

Answer 1

F1 is a suitable measure of models tested with imbalance datasets. But I think F1 is mostly a measure for models, rather than datasets. You could not say that dataset A is better than dataset B. There is no better or worse here; dataset is dataset.

Answer 2

I'm a complete noob and I'm not sure if my case translates to the case you're describing, but I'll have an attempt and accept criticism.

From my experience, the problem with F1-score is that it doesn't consider true-negatives. This means that in the case of heavily inbalanced datasets, the false-positives (when considering the minority class) will dominate, since we do not consider how big the proportion of false-positives is of all the negatives.

Consider a classifier predicting like this:

True-positives:  100
False-negatives:   0
False-positives: 100
F1-score: 0.667

Of all the positives, things are looking good, but the F1-score will punish this model by the relation between false-positives and true-positives.

But what if those 100 false-positives come from a set of 1000 negatives, which means the classifier has a 10% chance of predicting a false-positive - compared to if that is from a million negatives, in which case the chance is 0.0001%. The latter is more impressive. This is not a problem when considering models trained on the same datasets, but the F1-score requires a bit more context to understand what it translates to in terms of performance, and especially when comparing across datasets.

I believe the Matthews_correlation_coefficient solves the problem by considering all 4 elements of the confusion-matrix. If we plot the MCC-score as true-negatives approaches infinity, we see that MCC increases, since the proportion of false-positives gets smaller and smaller.

I still need to test the metric on more cases, and I don't know if this relation is only apparent with small numbers like in my example.

Answer 3

F1 score is used in the case where we have skewed classes i.e one type of class examples more than the other type class examples. Mainly we consider a case where we have more negative examples that positive examples. We calculate the F1 value by changing the threshold classifier value. The more the F1 values, the better it performs. In your case, data set B has higher F1 values. So, it will have better performance than A.

Answer 4

In my opinion, it always depends on the application when we say model A is better than model B.

I mean it’s biased to evaluate models according to a single metric, like F1 though it’s the combination of precision and recall. For example, in cases where you may want as many as instances of minority class been correctly classified, then you may use recall as the metric. In some other cases, you may want a higher precision among instances been classified as the minority class, then you can use precision.

Particularly, for class-imbalance data, AUC is a commonly used metric which takes care of the rank of positive class instances and negative ones. Take the minority class as the positive class, you may want to give higher confidence score to positive instances than negative ones. I personally think AUC is a more preferable metric than F1 in your context.

Hope it helps.

Answer 5

Along the lines of Jeppe's answer and my maths are correct, I disagree with the answers saying you can compare the performance of a model on two datasets using F-score.

Imagine these two cases:

Case 1:
        PredN  PredP
    N   20000  5000
    P   200    300

Case 2:
        PredN  PredP
    N   50000  5000
    P   200    300

These two confusion matrices have the same F-score:

Precision = TP/PP = 300/(300+5000) = 0.05660377358
Recall = TP/P = 300/(200+300) = 0.6
F-score = 2*Precision*Recall/(Precision+Recall) = 0.1034482759
 

However, in case 1, 5k ot of 20k negative classes are wrongly classified as positive. Meanwhile, in case 2, 5k out of 50k are misclassified. In some cases, misclassifying a 10% of the negatives is not the same as misclassifying 25% of them. In fact, imagine we had only 5000 negative cases and predicted all-positive (useless model), we'd still get that same Fscore.

This of course depends on your goal and costs of misclassifying each class. If the false positive rate is not a problem in your case, then this could not be a problem.

I like using the harmonic mean of recall and specificity instead although it does not seem as common.