6
$\begingroup$

I understand that AUC under ROC curve is a classic evaluation measurement for classifiers (which is basically the accuracy). However, when data is imbalanced, PR will be alternative. So, what does the AUC under PR curve mean?

Also, for example, if I obtain an optimal threshold for classifier in ROC curve (like the one that minimize the error), can I use that optimal threshold and calculate the Precision and Recall in PR-curve?

$\endgroup$
  • $\begingroup$ Careful: area under the ROC curve is not equivalent to accuracy. Area under the ROC curve is equivalent to concordance (aka $c$-statistic). This can be interpreted as the probability that a random positive is assigned a higher score than a random negative. See e.g. here. $\endgroup$ – Marc Claesen Nov 10 '14 at 10:11
7
$\begingroup$

Area under the ROC curve is equivalent to concordance (aka $c$-statistic) (not accuracy!). This can be interpreted as the probability that a random positive is assigned a higher score than a random negative. Unfortunately, area under the PR curve has no such interpretation (that I'm aware of).

The relationship between ROC and PR curves stems from the fact that both are based on the same source: contingency tables for every possible decision value threshold. Every threshold $T$ leads to a contingency table (e.g. $TP^{(T)}$, $FP^{(T)}$, $TN^{(T)}$, $FN^{(T)}$).

Every point in ROC space is based on a certain decision threshold $T$, and therefore coincides with a point in PR space. If a given model's ROC/PR curve dominates another, that model's PR/ROC curve will also dominate (cfr. Davis & Goadrich).

Also, for example, if I obtain an optimal threshold for classifier in ROC curve (like the one that minimize the error), can I use that optimal threshold and calculate the Precision and Recall in PR-curve?

Two remarks: if you want to select the threshold which minimizes the error (maximizes accuracy), ROC curves are not necessary (in fact they don't even show that). Secondly, if you have decided on a threshold you can just use the corresponding contingency table to get whatever other measures you want directly. Don't bother computing a full PR curve to then select 1 point of it.

Keep in mind that neither ROC or PR curves show you which threshold yields a certain point in the given space. They just show you the possible tradeoffs the model is capable of. That said, you can obviously map a point in ROC/PR space to a threshold if you retain a record of what thresholds they correspond to (most software packages to do this anyway).

$\endgroup$
  • $\begingroup$ Great answer, so I have one thing to clarify for second question. So, do you mean most algorithm do find the optimal threshold (minimize the error, that is maximize the accuracy) and then with that threshold I can calculate the corresponding F1-score (harmonice mean of precision and recall). Also, so what does PR-curve used for? $\endgroup$ – RockTheStar Nov 11 '14 at 19:49
  • 2
    $\begingroup$ @RockTheStar each learning algorithm tries to optimize a certain cost function during training. This includes selection of the threshold when applicable. Some learning algorithms try to minimize the error rate and will select the threshold based on that goal. Depending on your application, the cost function of the learning algorithm may or may not reflect what you actually want. If it does, great, you can stop after training. Otherwise, you can use ROC/PR/lift/... curves to select another threshold that more closely resembles your requirements. $\endgroup$ – Marc Claesen Nov 12 '14 at 8:19
  • $\begingroup$ Great responses! When you mention some learning algorithms, what are they? Thanks $\endgroup$ – RockTheStar Nov 12 '14 at 18:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.