23
$\begingroup$

I've used h2o.glm() function in R which gives a contingency table in the result along with other statistics. The contingency table is headed "Cross Tab based on F1 Optimal Threshold"

Wikipedia defines F1 Score or F Score as the harmonic mean of precision and recall. But aren't Precision and Recall found only when the result of predicted values of a logistic regression(for example) is transformed to binary using a cutoff.

Now by cutoff I remember, what is the connection between F1 Score and Optimal Threshold. How is optimal threshold calculated? How is F1 optimal threshold calculated?

Sorry if I've missed something, I'm new to stats here.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
34
$\begingroup$

I actually wrote my first paper in machine learning on this topic. In it, we identified that when your classifier outputs calibrated probabilities (as they should for logistic regression) the optimal threshold is approximately 1/2 the F1 score that it achieves. This gives you some intuition. The optimal threshold will never be more than .5. If your F1 is .5 and the threshold is .5, then you should expect to improve F1 by lowering the threshold. On the other hand, if the F1 were .5 and the threshold were .1, you should probably increase the threshold to improve F1.

The paper with all details and a discussion of why F1 may or may not be a good measure to optimize (in both single and multilabel case) can be found here:

https://arxiv.org/abs/1402.1892

Sorry that it took 9 months for this post to come to my attention. Hope that you still find the information useful!

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ Can F1 be >1? If you have 90% A, & 10% ~A, I would think you'd want a threshold >.5. $\endgroup$
    – gung - Reinstate Monica
    Commented Apr 14, 2016 at 18:53
  • 1
    $\begingroup$ Hi @gung. No, by definition F1 = 2*p*r/(p+r) and, like all F-beta measures, has range [0,1]. Class imbalance does not change the range of F1 score. For some applications, you may indeed want predictions made with a threshold higher than .5. Specifically, this would happen whenever you think false positives are worse than false negatives. But such a threshold wouldn't optimize F1 score. To understand why, F1 score was developed in the context of information retrieval. In these settings, the positive class is rare and typically false positives are not as costly as false negatives. $\endgroup$
    – Zachary Chase Lipton
    Commented Apr 15, 2016 at 2:56
  • $\begingroup$ @ZacharyChaseLipton Assume I have a dataset split into train/val/test. For a classifier that outputs a probability I would select the optimal F1 threshold on the validation set by examining the threshold that yields the best F1. This seems reasonable as selecting the threshold seems similar to selecting the best model. Is that the correct thing to do? $\endgroup$
    – pir
    Commented Jun 7, 2017 at 0:34
  • $\begingroup$ Moreover, assume I have a classifier that does not output probabilities (like an SVM). How would you optimize the F1 on the validation set then? $\endgroup$
    – pir
    Commented Jun 7, 2017 at 0:35
  • $\begingroup$ I've made it into a question: stats.stackexchange.com/questions/283931/… $\endgroup$
    – pir
    Commented Jun 7, 2017 at 0:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.