0
$\begingroup$

Can someone explain in an intuitive way the difference between Average_Precision_Score and AUC?

I read the documentation and understand that they are calculated slightly differently. But what is the real difference?

Also Average_Precision_Score is calculated - if I am correct - in terms of Recall over Precision. Not sure I understand.

Thanks in advance.

$\endgroup$
3
$\begingroup$

AUC (or AUROC, area under receiver operating characteristic) and AUPR (area under precision recall curve) are threshold-independent methods for evaluating a threshold-based classifier (i.e. logistic regression). Average precision score is a way to calculate AUPR. We'll discuss AUROC and AUPRC in the context of binary classification for simplicity.

Tl;dr:

  • The ROC is a curve that plots true positive rate (TPR) against false positive rate (FPR) as your discrimination threshold varies.
  • AUROC is the area under that curve (ranging from 0 to 1); the higher the AUROC, the better your model is at differentiating the two classes.
  • AUPRC is the area under the precision-recall curve, which similarly plots precision against recall at varying thresholds.
  • sklearn.metrics.average_precision_score gives you a way to calculate AUPRC.

On AUROC

The ROC curve is a parametric function in your threshold $T$, plotting false positive rate (a.k.a. 1 - specificity, usually on x-axis) versus true positive rate (a.k.a. recall, on y-axis). Intuitively, this metric tries to answer the question "as my decision threshold varies, how well can my classifier discriminate between negative + positive examples?" In fact, AUROC is statistically equivalent to the probability that a randomly chosen positive instance will be ranked higher than a randomly chosen negative instance (by relation to the Wilcoxon rank test -- I don't know the details of the proof though).

Here's a nice schematic that illustrates some of the core patterns to know: Diagram of varying ROC curves.

  • Red dotted line: As you can see, a random classifier will have an AUROC of 0.5. To see why, think about what happens when you choose a threshold such that 60% of all points are randomly predicted positive; then try to reason about the expected TPR/FPR in that case.
  • Green, blue, and orange lines: The AUROC increases for each of these curves as they "swell" away from the red dotted line, towards the blue dotted point. These classifiers get better and better; as at varying thresholds, the TPR consistently "outpaces" the FPR.

For further reading -- Section 7 of this is highly informative, which also briefly covers the relation between AUROC and the Gini coefficient. You can also find a great answer for an ROC-related question here. Lastly, here's a (debatable) rule-of-thumb for assessing AUROC values: 90%—100%: Excellent, 80%—90%: Good, 70%—80%: Fair, 60%—70%: Poor, 50%—60%: Fail.

On AUPR

One of the key limitations of AUROC becomes most apparent on highly imbalanced datasets (low % of positives, lots of negatives), e.g. many medical datasets, rare event detection problems, etc. Small changes in the number of false positives/false negatives can severely shift AUROC. AUPR, which plots precision vs. recall parametrically in threshold $t$ (similar setup to ROC, except the variables plotted), is more robust to this problem.

The baseline value for AUPR is equivalent to the ratio of positive instances to negative instances; i.e. $\left(\frac{\#(+)}{\#(-)\; + \;\#(+)}\right)$. Similarly to AUROC, this metric ranges from 0 to 1, and higher is "better."

Now, to address your question about average precision score more directly, this gives us a method of computing AUPR using rectangles somewhat reminiscent of Riemannian summation (without the limit business that gives you the integral). Average precision score gives us a guideline for fitting rectangles underneath this curve prior to summing up the area. Let's say that we're doing logistic regression and we sample 11 thresholds: $T = \{0.0, 0.1, 0.2, \dots, 1.0\}$. Each threshold $t_n$ is going to give us a corresponding value of precision and recall $P_n, R_n$; we can plot each of those and connect the dots. Perhaps we end up with a curve like the one we see below. The width of the rectangle is the difference in recall achieved at the $n$th and $n-1$st threshold; the height is the precision achieved at the $n$th threshold. You can easily see from the step-wise shape of the curve how one might try to fit rectangles underneath the curve to compute the area underneath.

On a related note, yes, you can also squish trapezoids underneath the curve (this is what sklearn.metrics.auc does) -- think about what advantages/disadvantages might occur in that case.

enter image description here

For further reading, I found this to be a nice resource for showing the limitations of AUROC in favor of AUPR in some cases.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.