6
$\begingroup$

I have a precision recall curve for two separate algorithms. If I want to calculate the F-Measure I have to use the precision and recall values at a particular point on each curve.

How is this point decided? For example on curve one there is a point where recall is 0.9 and precision is 0.87 and the other curve there is a point of recall at 0.95 and precision at 0.84.

Alternatively, should I plot a F-measure curve for every precision recall value?

$\endgroup$
  • $\begingroup$ I think this has been discussed before in the setting of AUC. In this analogue F-measure=Youden Index. $\endgroup$ – charles Apr 8 '14 at 19:03
  • $\begingroup$ When you say precision/recall curve are you referring to ROCs where sensitivity/specificity are calculated over a range of thresholds? Youden's and F-scores are functions of a single sensitivity/specificity. You would have to choose a single threshold and report values for that threshold. $\endgroup$ – AdamO Apr 8 '14 at 21:18
  • $\begingroup$ Regarding your question "How is this point decided?": I think an appropriate choice would be the EER (equal error rate). This is the point where TPR and FPR cross. Have a look on this: security.stackexchange.com/questions/57589/… $\endgroup$ – Unhandled exception Jul 29 '15 at 15:12
2
$\begingroup$

It is hard to read off F1 (or any other weighted F-measure) directly from a Precision-Recall graph, because of needing to work with reciprocals (harmonic mean).

But if instead you plot the reciprocal Precision & Recall, then values of the F-measure form isobars (straight lines with equal values) with gradient depending on the tradeoff parameter. In the case of F1 they will be isobars parallel to the diagonal, corresponding to equal weighting of success in terms of precision of the positive predictions and in terms of recall of the positive cases (the number predicted may not correspond to the real number of positives).

If you use isobars directly on the Precision-Recall graph, you optimize the arithmetic mean instead - the advantage of using harmonic over arithmetic is debatable and is discussed in the reference, and there is some evidence that geometric is better than either.

If you plot Precision & Recall logarithmically, then the isobars can be used to optimize this geometric mean.

Note that the logarithmic and reciprocal PR graphs are not defined at the points corresponding to no positives are present or none are predicted. At all other points these curves are equivalent to the ROC curve in that if a solution is better ranked in one it will be better ranked in the other.

ROC curves are however a bit different as the compare TPR (Recall) against FPR (Fallout) but again isobars are useful and those parallel to the diagonal correspond to equal weighting of positives and negatives. In particular Precision and Recall are independent of the number of True Negatives (correctly predicted negatives) but this is a complmenetary component of Fallout - in fact a mirror image graph is formed by plotting TPR vs TNR. The difference TPR-FPR is also known as Youden J or Informedness, and is linearly related to the area under the curve formed by a specific operating point.

This is discussed in detail in a report of mine I have uploaded, with the graphs I've discussed shown in Figure 3 for some real data relating to facial expression recognition: https://www.researchgate.net/publication/273761103_What_the_F-measure_doesn%27t_measure

Discussion of ROC Area Under the Curve is here: https://www.researchgate.net/publication/261155937_The_problem_of_Area_Under_the_Curve?ev=pub_cit

$\endgroup$
1
$\begingroup$

Precision-Recall curve and ROC curve (doesn't matter they are just the mirror images of each other) are used to give you the sense of the quality of the binary classifier for the different values for some parameter that affects the performance of your classifier. Now, F1 are particular scores which combine both precision and recall into a single one, so that way you just need to select the configuration of your classifier which has the highest F score.

In your place for each pair of precision and recall I would calculate F score and then pick the configuration which has the highest F score.

Now, the tricky part is which F score. F1 is the score which values precision and recall the same, but sometimes the recall is more important than precision (for example, you don't mind having a lot of people falsely tested for some cancer if you know that all of the ones who have that cancer are tested). In that case you could use F2 measure.

I think it doesn't make sense to sum up all F measures for all combinations of precision and recall. After all, the idea is to pick a single model out of the broader range of models, I would prefer to pick a model with the highest value of F score instead of the one with the biggest sum of all F scores.

$\endgroup$
  • 4
    $\begingroup$ PR and ROC curves are completely different, not mirror images. PR doesn't depend on the number of True Negative but ROC does. PR isn't always monotonic but ROC is. $\endgroup$ – David M W Powers Mar 26 '15 at 3:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.