Comparing probability threshold graphs for F1 score for different models

Question

Below are two plots, side-by side, for an imbalanced dataset.

We have a very large imbalanced dataset that we are processing/transforming in different manner. After each transformation, we run an xgboost estimator over it.

On the left are the PR-curves of three xgboost models over three differently transformed datasets. As can be seen from the left diagram, all the three PR curves overlap; and in fact, areas under the curve of two of them (red and green) are the same.

On the right are the plots from the same three models but of F1 score (calculated over test data) at various probability thresholds. Colors of models in both left and right plots match. Peak F1 score, at different probability thresholds, is about the same for both red and green models. Peak F1 score for blue model is a little less than the peak F1 score of the two other models. My questions are:

a. Can I say, model green is better than model red as its F1 score is quite stable over a large range of probability thresholds, while that for red model F1 score falls rapidly with a little change in probability threshold.
b. Of the two, models red and blue which one is better and why?

Briefly, the question relates to interpreting F1-score threshold graphs of different models, given that PR curves lead to no conclusion.

I will be grateful for a reply. Incidentally I have gone through a large number of discussions on F1 score, AUC and PR curves including this one.

Answer 1

The three models are likely almost the same, just with a monotonic shift of their predicted probabilities (see King and Zeng 2001, and DS.SE "Is There a Way to Re-Calibrate Predicted Probabilities After Using Class Weights?"). In the left plot, this manifests as almost-overlapping PR curves, but you will find that the thresholds defining each point are different (shifted). In the right plot, it manifests as transforming (nonlinearly) the horizontal axis for the curves. Note too that since F1 is defined in terms of precision and recall, having (almost) the same PR curves means you will have (almost) the same F1, just perhaps at different thresholds.

The largest effect of resampling/weighting schemes is to apply a linear shift to the predicted log-odds. This is rigorously shown for logistic regression (under certain assumptions; resampling shifts the intercept term while coefficients remain unbiased estimates of the originals), and in my experience holds for other models as well (see that DS.SE post again for an experiment, but I've seen it work in other settings too). See also Are unbalanced datasets problematic, and (how) does oversampling (purport to) help? and its linked questions, and Does oversampling/undersampling change the distribution of the data?

These approaches can improve predictions in other ways, but it's fairly rare AFAICT, and not nearly as pronounced as the shift in probabilities. See e.g.
How does class balancing via reweighting affect logistic regression?
Is up- or down-sampling imbalanced data actually that effective? Why?

Now, SMOTE and similar approaches that aren't purely resampling may provide some further differences, but I haven't seen any convincing studies of that yet. At any rate, your PR curves show that it's not changing the models significantly in terms of precision and recall.

So:

a. Can I say, model green is better than model red as its F1 score is quite stable over a large range of probability thresholds, while that for red model F1 score falls rapidly with a little change in probability threshold.

No, it's just that the "good" thresholds are more squished (by the nonlinear transformation) toward zero for the red model. (The spikiness of the red curve might warrant further investigation.)

b. Of the two, models red and blue which one is better and why?

I don't think you can tell from these, but if one is better it's not by much. The red curve presumably better represents true probabilities of the events, so if calibration is important to you, go with that one, or undo the shift as post-processing of the others.