10
$\begingroup$

Because I have a very imbalanced dataset (9% positive outcomes), I decided a precision-recall curve was more appropriate than an ROC curve. I obtained the analogous summary measure of area under the P-R curve (.49, if you're interested) but am unsure of how to interpret it. I've heard that .8 or above is what a good AUC for ROC is, but would the general cutoffs be the same for the AUC for a precision-recall curve?

$\endgroup$
10
$\begingroup$

There is no magic cut-off for either AUC-ROC or AUC-PR. Higher is obviously better, but it is entirely application dependent.

For example, if you could successfully identify profitable investments with an AUC of 0.8 or, for that matter anything distinguishable from chance, I would be very impressed and you would be very rich. On the other hand, classifying handwritten digits with an AUC of 0.95 is still substantially below the current state of the art.

Furthermore, while the best possible AUC-ROC is guaranteed to be in [0,1], this is not true for precision-recall curves because there can be "unreachable" areas of P-R space, depending on how skewed the class distributions are. (See this paper by Boyd et al (2012) for details).

$\endgroup$
  • $\begingroup$ I thought there we unreachable parts of AUC as well. But could be wrong. $\endgroup$ – charles Aug 26 '14 at 19:59
  • 4
    $\begingroup$ The paper I linked says "A related, but previously unrecognized, distinction between the two types of curves is that, while any point in ROC space is achievable, not every point in PR space is achievable." at the top of page 2. I think it's because you must rank all the documents in your collection for P/R, so even the most pessimistic system will eventually retrieve a relevant item. For ROC though, you could call all the positive examples "-" and all the negative examples "+", which would give you a 100% false positive/100% false negative rate. $\endgroup$ – Matt Krause Aug 26 '14 at 20:44
  • $\begingroup$ Thanks! I should have looked at paper before commenting. $\endgroup$ – charles Aug 26 '14 at 23:03
  • $\begingroup$ I agree that there is no magic number. However, there is definitely value in understanding that a 0.95 AUC-ROC, for example, means that you have essentially solved the problem and have a very, very good classifier. Whereas an AUC of 0.6 for finding profitable investments might be, strictly speaking, better than random, but not much better. That said, as you mentioned, it would still be supposedly distinguishable from chance, and might well yield you a nice strategy. $\endgroup$ – shiri Dec 10 '18 at 16:15
0
$\begingroup$

.49 is not great, but its interpretation is different than the ROC AUC. For ROC AUC, if you obtained a .49 using a logistic regression model, I would say you are doing no better than random. For .49 PR AUC, however it might not be that bad. I would consider looking at individual precision and recall, perhaps one or the other is what is driving down your PR AUC. Recall will tell you how much of that 9% positive class you are actually guessing correct. Precision will tell you how many you guessed positive that were not. (False Positives). 50% recall would be bad meaning you're not guessing many of your imbalanced class, but perhaps 50% precision wouldn't bad. Depends on your situation.

$\endgroup$
0
$\begingroup$

A random estimator would have a PR-AUC of 0.09 in your case (9% positive outcomes), so your 0.49 is definitely a substantial increase.

If this is a good result could only be assessed in compariso to other algorithms, but you didn't give detail on the method/data you used.

Additionally, you might want to assess the shape of your PR-curve. An ideal PR-curve goes from the topleft corner horizontically to the topright corner and straight down to the bottomright corner, resulting in a PR-AUC of 1. In some applications, the PR-curve shows instead a strong spike at the beginning to quickly drop again close to the "random estimator line" (the horizontal line at 0.09 precision in your case). This would indicate a good detection of "strong" positive outcomes, but poor performance on the less clear candidates.

If you want to find a good threshold for your algorithm's cutoff parameter, you might consider the point on the PR-curve that's closest to the topright corner. Or even better, consider cross validation if possible. You might achieve precision and recall values for a specific cutoff parameter that are more interesting for your application than the value of the PR-AUC. The AUCs are most interesting when comparing different algorithms.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.