13
$\begingroup$

I understand that AUC under ROC curve is a classic evaluation measurement for classifiers (which is basically the accuracy). However, when data is imbalanced, PR will be alternative. So, what does the AUC under PR curve mean?

Also, for example, if I obtain an optimal threshold for classifier in ROC curve (like the one that minimize the error), can I use that optimal threshold and calculate the Precision and Recall in PR-curve?

Improve this question
$\endgroup$
1
  • 3
    $\begingroup$ Careful: area under the ROC curve is not equivalent to accuracy. Area under the ROC curve is equivalent to concordance (aka $c$-statistic). This can be interpreted as the probability that a random positive is assigned a higher score than a random negative. See e.g. here. $\endgroup$
    – Marc Claesen
    Nov 10, 2014 at 10:11

1 Answer 1

Reset to default
16
$\begingroup$

Area under the ROC curve is equivalent to concordance (aka $c$-statistic) (not accuracy!). This can be interpreted as the probability that a random positive is assigned a higher score than a random negative. Unfortunately, area under the PR curve has no such interpretation (that I'm aware of).

The relationship between ROC and PR curves stems from the fact that both are based on the same source: contingency tables for every possible decision value threshold. Every threshold $T$ leads to a contingency table (e.g. $TP^{(T)}$, $FP^{(T)}$, $TN^{(T)}$, $FN^{(T)}$).

Every point in ROC space is based on a certain decision threshold $T$, and therefore coincides with a point in PR space. If a given model's ROC/PR curve dominates another, that model's PR/ROC curve will also dominate (cfr. Davis & Goadrich).

Also, for example, if I obtain an optimal threshold for classifier in ROC curve (like the one that minimize the error), can I use that optimal threshold and calculate the Precision and Recall in PR-curve?

Two remarks: if you want to select the threshold which minimizes the error (maximizes accuracy), ROC curves are not necessary (in fact they don't even show that). Secondly, if you have decided on a threshold you can just use the corresponding contingency table to get whatever other measures you want directly. Don't bother computing a full PR curve to then select 1 point of it.

Keep in mind that neither ROC or PR curves show you which threshold yields a certain point in the given space. They just show you the possible tradeoffs the model is capable of. That said, you can obviously map a point in ROC/PR space to a threshold if you retain a record of what thresholds they correspond to (most software packages to do this anyway).

Improve this answer
$\endgroup$
5
  • $\begingroup$ Great answer, so I have one thing to clarify for second question. So, do you mean most algorithm do find the optimal threshold (minimize the error, that is maximize the accuracy) and then with that threshold I can calculate the corresponding F1-score (harmonice mean of precision and recall). Also, so what does PR-curve used for? $\endgroup$
    – RockTheStar
    Nov 11, 2014 at 19:49
  • 2
    $\begingroup$ @RockTheStar each learning algorithm tries to optimize a certain cost function during training. This includes selection of the threshold when applicable. Some learning algorithms try to minimize the error rate and will select the threshold based on that goal. Depending on your application, the cost function of the learning algorithm may or may not reflect what you actually want. If it does, great, you can stop after training. Otherwise, you can use ROC/PR/lift/... curves to select another threshold that more closely resembles your requirements. $\endgroup$
    – Marc Claesen
    Nov 12, 2014 at 8:19
  • $\begingroup$ Great responses! When you mention some learning algorithms, what are they? Thanks $\endgroup$
    – RockTheStar
    Nov 12, 2014 at 18:00
  • 1
    $\begingroup$ @MarcClaesen You mentioned the interpretation of AUC for a ROC curve, but the OP was asking for an interpretation of AUC for a PR curve. Is the interpretation of AUC-PR similar (if not identical) to AUC-ROC? For example, the AUC-PR is the probability that a random positive is assigned a higher score than a random negative? $\endgroup$
    – Jane Wayne
    Aug 8, 2019 at 15:54
  • $\begingroup$ @JaneWayne AUC-PR is identical to average precision, see here. $\endgroup$
    – Eike P.
    Jan 1 at 17:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.