Sklearn Average_Precision_Score vs. AUC

Question

Can someone explain in an intuitive way the difference between Average_Precision_Score and AUC?

I read the documentation and understand that they are calculated slightly differently. But what is the real difference?

Also Average_Precision_Score is calculated - if I am correct - in terms of Recall over Precision. Not sure I understand.

Thanks in advance.

Answer 1

AUC (or AUROC, area under receiver operating characteristic) and AUPR (area under precision recall curve) are threshold-independent methods for evaluating a threshold-based classifier (i.e. logistic regression). Average precision score is a way to calculate AUPR. We'll discuss AUROC and AUPRC in the context of binary classification for simplicity.

Tl;dr:

• The ROC is a curve that plots true positive rate (TPR) against false positive rate (FPR) as your discrimination threshold varies.
• AUROC is the area under that curve (ranging from 0 to 1); the higher the AUROC, the better your model is at differentiating the two classes.
• AUPRC is the area under the precision-recall curve, which similarly plots precision against recall at varying thresholds.
• sklearn.metrics.average_precision_score gives you a way to calculate AUPRC.

On AUROC

The ROC curve is a parametric function in your threshold $T$, plotting false positive rate (a.k.a. 1 - specificity, usually on x-axis) versus true positive rate (a.k.a. recall, on y-axis). Intuitively, this metric tries to answer the question "as my decision threshold varies, how well can my classifier discriminate between negative + positive examples?" In fact, AUROC is statistically equivalent to the probability that a randomly chosen positive instance will be ranked higher than a randomly chosen negative instance (by relation to the Wilcoxon rank test -- I don't know the details of the proof though).

Here's a nice schematic that illustrates some of the core patterns to know:

• Red dotted line: As you can see, a random classifier will have an AUROC of 0.5. To see why, think about what happens when you choose a threshold such that 60% of all points are randomly predicted positive; then try to reason about the expected TPR/FPR in that case.
• Green, blue, and orange lines: The AUROC increases for each of these curves as they "swell" away from the red dotted line, towards the blue dotted point. These classifiers get better and better; as at varying thresholds, the TPR consistently "outpaces" the FPR.

For further reading -- Section 7 of this is highly informative, which also briefly covers the relation between AUROC and the Gini coefficient. You can also find a great answer for an ROC-related question here. Lastly, here's a (debatable) rule-of-thumb for assessing AUROC values: 90%—100%: Excellent, 80%—90%: Good, 70%—80%: Fair, 60%—70%: Poor, 50%—60%: Fail.

On AUPR

One of the key limitations of AUROC becomes most apparent on highly imbalanced datasets (low % of positives, lots of negatives), e.g. many medical datasets, rare event detection problems, etc. Small changes in the number of false positives/false negatives can severely shift AUROC. AUPR, which plots precision vs. recall parametrically in threshold $t$ (similar setup to ROC, except the variables plotted), is more robust to this problem.

The baseline value for AUPR is equivalent to the ratio of positive instances to negative instances; i.e. $\left(\frac{\#(+)}{\#(-)\; + \;\#(+)}\right)$. Similarly to AUROC, this metric ranges from 0 to 1, and higher is "better."

Now, to address your question about average precision score more directly, this gives us a method of computing AUPR using rectangles somewhat reminiscent of Riemannian summation (without the limit business that gives you the integral). Average precision score gives us a guideline for fitting rectangles underneath this curve prior to summing up the area. Let's say that we're doing logistic regression and we sample 11 thresholds: $T = \{0.0, 0.1, 0.2, \dots, 1.0\}$. Each threshold $t_n$ is going to give us a corresponding value of precision and recall $P_n, R_n$; we can plot each of those and connect the dots. Perhaps we end up with a curve like the one we see below. The width of the rectangle is the difference in recall achieved at the $n$th and $n-1$st threshold; the height is the precision achieved at the $n$th threshold. You can easily see from the step-wise shape of the curve how one might try to fit rectangles underneath the curve to compute the area underneath.

On a related note, yes, you can also squish trapezoids underneath the curve (this is what sklearn.metrics.auc does) -- think about what advantages/disadvantages might occur in that case.

For further reading, I found this to be a nice resource for showing the limitations of AUROC in favor of AUPR in some cases.