6
$\begingroup$

From what I understand, one can use the AUC of the ROC curve as a summary statistic of the full curve.

Q1. Are there any similar summary statistics that one can use on a single precision-recall curve?

Q2. As far as I understand, the $F$-score ($F_1$ or $F_\beta$), is measured at a specific operating precision-recall regime. i.e. one needs to fix a point on the precision-recall curve to obtain a precision rate and it's associated recall rate to obtain a result in the formulas:

$F_1 = 2 \cdot \frac{\mathrm{precision} \cdot \mathrm{recall}}{\mathrm{precision} + \mathrm{recall}}$

$F_\beta = (1 + \beta^2) \cdot \frac{\mathrm{precision} \cdot \mathrm{recall}}{(\beta^2 \cdot \mathrm{precision}) + \mathrm{recall}}$

If so, how does one choose such point on the curve?

$\endgroup$
5
$\begingroup$

The "Mean Average Precision" (sometimes abbreviated mAP or MAP) might be what you want. It's pretty commonly used for evaluating information retrieval systems and is fairly straightforward to compute.

First, calculate the average precision for a given query. To do this, rank the documents and compute the precision after retrieving each relevant document. For example, suppose that four documents are relevant to this query, and our system returned the following:

  1. Relevant document
  2. Irrelevant document
  3. Relevant document
  4. Relevant document
  5. Irrelevant document
  6. Irrelevant document.
  7. Relevant document

The first relevant document is at position one, and the precision there is 1/1 = 1.0 The next relevant document is at position 3; two of the three documents seen so far are relevant, so our precision here is 2/3. Document 4 is relevant too and the precision score here is 3/4. The final relevant item is at position seven, giving us a precision of 4/7.

Find the mean of these precision scores (1/4*(1 + 2/3 + 3/4 + 4/7) = ~0.747) to get the average precision for this query. The mean average precision is just the mean of these averages across all the queries in your evaluation set.

As for choosing a precision-recall trade off, that's largely up to you. The $F_1$ score gives them equal weight; you can interpret the $\beta$ in $F_\beta$ as giving $\beta$ times more weight to recall than precision. I believe that some studies indicate that users prefer precision to recall, but I would bet that it depends a lot on the application and use-case. I certainly don't need google to show me every webpage about cats, but do want all the sites on the first page to be relevant. On the flip side, it might be more important to return every possibly-relevant document if you're doing discovery for a court case.

$\endgroup$
  • $\begingroup$ This is a great answer. Thanks Matt! One question that I still have is when you said "$F_1$ gives them [precision - recall] equal weight". But you still have to choose a point on the prec-recall curve to fill in the numbers in the $F_1$ formula, correct? $F_1$ gives them equal weight, but I may choose a point on the curve where precision is higher than recall (or the other way around). Is my understanding right? $\endgroup$ – Amelio Vazquez-Reina Feb 14 '13 at 14:37
  • $\begingroup$ Yup! You could think about it like this: choosing the precision-recall trade-off is just like any other parameter of your system. I was trying to hint that you could either 1) vary your selection of that point OR 2) vary your choice of $\beta$ for the $F_\beta$ scores if you wanted to report several of them. $\endgroup$ – Matt Krause Feb 14 '13 at 20:22
4
$\begingroup$

Actually there is just an AUC of PR-curve measure; it is used in biology (especially in the DREAM challenge series surroundings) because it is consistent with AUROC (i.e. the ranking is usually the same if the performance differs significantly) still gives better numerical resolution by giving lower values than AUROC.

The problem is that AUPR requires careful integration, so it is pretty hard to find a correct implementation.

This is a canonical paper about the topic.

$\endgroup$
  • $\begingroup$ Thanks mbq. One question that I still have and that I am not sure @Matt has already answered is: Do I need to pick a specific point on the prec-recall curve to fill in the numbers in the $F_1$ formula? (otherwise how do I compute it's value?). If so, how do I pick that point? (and specific precision recall numbers)? $\endgroup$ – Amelio Vazquez-Reina Feb 14 '13 at 19:03
2
$\begingroup$

You can calculate the AUC of the ROC for just a single (precision,recall) data point.

This paper, Robust classification for imprecise environments, describes how to calculate the convex hull AUC (which is pretty standard now). When you have only one (precision,recall) point, you extend a straight line down to the always-say-no (0,0) point and a straight line up to the always-say-yes (1,1) point, and you have the convex hull.

Now the neat result: in this case, with only one coordinate, the calculation simplifies to

$AUC = (t − f + 1)/2 $.

This emphasises the connection between AUC and the Gini coefficient, remarked on elsewhere.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.