12
$\begingroup$

I've used h2o.glm() function in R which gives a contingency table in the result along with other statistics. The contingency table is headed "Cross Tab based on F1 Optimal Threshold"

Wikipedia defines F1 Score or F Score as the harmonic mean of precision and recall. But aren't Precision and Recall found only when the result of predicted values of a logistic regression(for example) is transformed to binary using a cutoff.

Now by cutoff I remember, what is the connection between F1 Score and Optimal Threshold. How is optimal threshold calculated? How is F1 optimal threshold calculated?

Sorry if I've missed something, I'm new to stats here.

$\endgroup$
19
$\begingroup$

I actually wrote my first paper in machine learning on this topic. In it, we identified that when your classifier outputs calibrated probabilities (as they should for logistic regression) the optimal threshold is approximately 1/2 the F1 score that it achieves. This gives you some intuition. The optimal threshold will never be more than .5. If your F1 is .5 and the threshold is .5, then you should expect to improve F1 by lowering the threshold. On the other hand, if the F1 were .5 and the threshold were .1, you should probably increase the threshold to improve F1.

The paper with all details and a discussion of why F1 may or may not be a good measure to optimize (in both single and multilabel case) can be found here:

https://arxiv.org/abs/1402.1892

Sorry that it took 9 months for this post to come to my attention. Hope that you still find the information useful!

$\endgroup$
  • 1
    $\begingroup$ Can F1 be >1? If you have 90% A, & 10% ~A, I would think you'd want a threshold >.5. $\endgroup$ – gung Apr 14 '16 at 18:53
  • $\begingroup$ Hi @gung. No, by definition F1 = 2*p*r/(p+r) and, like all F-beta measures, has range [0,1]. Class imbalance does not change the range of F1 score. For some applications, you may indeed want predictions made with a threshold higher than .5. Specifically, this would happen whenever you think false positives are worse than false negatives. But such a threshold wouldn't optimize F1 score. To understand why, F1 score was developed in the context of information retrieval. In these settings, the positive class is rare and typically false positives are not as costly as false negatives. $\endgroup$ – Zachary Chase Lipton Apr 15 '16 at 2:56
  • $\begingroup$ @ZacharyChaseLipton Assume I have a dataset split into train/val/test. For a classifier that outputs a probability I would select the optimal F1 threshold on the validation set by examining the threshold that yields the best F1. This seems reasonable as selecting the threshold seems similar to selecting the best model. Is that the correct thing to do? $\endgroup$ – pir Jun 7 '17 at 0:34
  • $\begingroup$ Moreover, assume I have a classifier that does not output probabilities (like an SVM). How would you optimize the F1 on the validation set then? $\endgroup$ – pir Jun 7 '17 at 0:35
  • $\begingroup$ I've made it into a question: stats.stackexchange.com/questions/283931/… $\endgroup$ – pir Jun 7 '17 at 0:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.