Area under the F1 curve

Question

Since the F1-score is threshold-dependent, would the area under F1 across all thresholds be an informative metric?

Answer 1

Yes, it would be informative, but not massively.

A bit of context: The F1 curve is more often referred to as the "F1 confidence curve" and it is a pretty specialised visualisation. Its AUC is a really niche metric that I have mostly encountered in computer vision papers (e.g. Towards Machine Vision for Insect Welfare Monitoring and Behavioural Insights Hansen et al., BdSpell: A YOLO-based Real-time Finger Spelling System for Bangla Sign Language by Haque et al., etc.) These curves typically have a clear concave shape because, at both very low and very high thresholds, the F1 score will likely be (very) poor.

While higher AUC values for the F1 confidence curve indicate better performance across different thresholds, we already have more established alternatives for this purpose: the ROC curve and the PR curve. These alternatives not only provide similar information but also allow for direct performance comparisons between different methods. For that reason, unless the F1 confidence curve (and is AUC) is widely adopted in your specific field, it might be best treated as a supplementary visualisation in an Appendix. Yes, it can highlight whether small changes in the threshold significantly impact the F1 score, but its direct utility beyond that is quite limited.

Answer 2

The F1 curve

The F1-confidence curve describes the relationship between

• F1 score, which is a measure that combines precision and recall $$F_1 = \frac{2}{\frac{1}{\text{precision}} + \frac{1}{\text{recall}}}$$

• and the confidence threshold used in yolo algorithms for image detection described in:

  Redmon, J. "You only look once: Unified, real-time object detection." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016. arXiv:1506.02640

  This confidence, a combination of the probability of an item or specific class with an estimate for the accuracy of the bounding box.

  It is comparable to the probability parameter in the simpler and more well known binomial regression. Note that this probability or confidence is not the same as the probability of a correct classification. See : Probability threshold in ROC curve analyses

Finding the optimal algorithm/classifier

As you change the confidence threshold you will have to balance between

• high thresholds leading to low recall as many of the true positive cases will not be classified positive.
• low thresholds leading to low precision as many of the false positive cases will be falsely classified as positives.

This problem has multiple variables that are relevant. A single measure can not capture this.

Typically you will need to use some cost function to optimize your problem. How much do you loose or gain as function of an increase or decrease in precision and recall. The optimum is what is relevant and not some average over all possible confidence levels. This relates to the question: Is higher AUC always better?

Why AUC?

A reason to use AUC anyways, is in research where different classifiers are compared without a specific goal/cost-function in mind. In that case some sort of average like a AUC can be used. It is a limitation, but a multidimensional comparison is difficult and some people prefer a simple viewpoint like using some one-dimensional score and a ranking of classifiers.

Also, for ROC curves the AUC still has an intuitive probabilistic interpretation. For a random positive item $x_1$ and random negative item $x_2$ it is the probability that the confidence $p$ (or whatever other classification value) is larger for the positive item $P(p(x_1) > p(x_0))$. (I am not sure whether a AUC for a F1 curve has a similar interpretation, given the complexity I doubt it)