15
$\begingroup$

Is there any standard method to determine an "optimal" operation point on a precision recall curve? (i.e., determining the point on the curve that offers a good trade-off between precision and recall)

Thanks

Improve this question
$\endgroup$

4 Answers 4

Reset to default
16
$\begingroup$

The definition of "optimal" will of course depend on your specific goals, but here are a few relatively "standard" methods:

  • Equal error rate (EER) point: the point where precision equals recall. This feels to some people like a "natural" operating point.

  • A refined and more principled version of the above is to specify cost of the different kind of errors and optimize that cost. Say misclassifying an item (an error in precision) is twice as expensive as missing an item completely (error in recall). Then the best operating point is that where (1 - recall) = 2*(1 - precision).

  • In some problems people have a natural minimal acceptable rate of either precision or recall. Say you know that if more than 20% of retrieved data is incorrect, the users will stop using your application. Then it is natural to set precision to 80% (or a bit lower) and accept whatever recall you have at that point.

Improve this answer
$\endgroup$
2
$\begingroup$

Following up on SheldonCooper's second and third bullet points: The ideal choice is to have somebody else make the choice, either in the form of a threshold (point 3) or a cost benefit tradeoff (point 2). And perhaps the nicest way to offer them the choice is with an ROC curve.

Improve this answer
$\endgroup$
2
$\begingroup$

I'm not sure how "standard" this is, but one way would be to choose the point that is closest to (1, 1) -- i.e. 100% recall and 100% precision. That would be the optimal balance between the two measures. This is assuming you don't value precision over recall or vice-versa.

Improve this answer
$\endgroup$
0
$\begingroup$

Yes, the point closest to (1,1) is one way to choose an optimal threshold. This can be done systematically with recent publication such as LRP error or F1-score based method to select the optimal threshold.

  1. In the first method, the optimal confidence threshold is at which minimum LRP error is observed LRP Metric.
  2. In the second method, the optimal confidence threshold is at which maximum F1 score is observed F1-score based method.
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.