Because I have a very imbalanced dataset (9% positive outcomes), I decided a precision-recall curve was more appropriate than an ROC curve. I obtained the analogous summary measure of area under the P-R curve (.49, if you're interested) but am unsure of how to interpret it. I've heard that .8 or above is what a good AUC for ROC is, but would the general cutoffs be the same for the AUC for a precision-recall curve?
$\begingroup$
$\endgroup$
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
4
There is no magic cut-off for either AUC-ROC or AUC-PR. Obviously, higher is better, and the closer you are to 1.0, the closer you are to solving the problem.
However, the meaning of "close" is entirely application dependent. For example, if you could reliably identify profitable investments with an AUC of 0.7 or, for that matter anything distinguishable from chance, I would be very impressed and you would be very rich. On the other hand, classifying handwritten digits with an AUC of 0.95 is still substantially below the current state of the art.
Furthermore, while the best possible AUC-ROC is guaranteed to be in [0,1], this is not true for precision-recall curves because there can be "unreachable" areas of P-R space, depending on how skewed the class distributions are. This may render a "large" AUC-PR value less impressive than it might otherwise seem. See this paper by Boyd et al (2012) for details.
answered Aug 26, 2014 at 18:50
-
$\begingroup$ I thought there we unreachable parts of AUC as well. But could be wrong. $\endgroup$– charlesAug 26, 2014 at 19:59
-
5$\begingroup$ The paper I linked says "A related, but previously unrecognized, distinction between the two types of curves is that, while any point in ROC space is achievable, not every point in PR space is achievable." at the top of page 2. I think it's because you must rank all the documents in your collection for P/R, so even the most pessimistic system will eventually retrieve a relevant item. For ROC though, you could call all the positive examples "-" and all the negative examples "+", which would give you a 100% false positive/100% false negative rate. $\endgroup$ Aug 26, 2014 at 20:44
-
1$\begingroup$ Thanks! I should have looked at paper before commenting. $\endgroup$– charlesAug 26, 2014 at 23:03
-
$\begingroup$ I agree that there is no magic number. However, there is definitely value in understanding that a 0.95 AUC-ROC, for example, means that you have essentially solved the problem and have a very, very good classifier. Whereas an AUC of 0.6 for finding profitable investments might be, strictly speaking, better than random, but not much better. That said, as you mentioned, it would still be supposedly distinguishable from chance, and might well yield you a nice strategy. $\endgroup$– shiriDec 10, 2018 at 16:15
$\begingroup$
$\endgroup$
3
A random estimator would have a PR-AUC of 0.09 in your case (9% positive outcomes), so your 0.49 is definitely a substantial increase.
If this is a good result could only be assessed in compariso to other algorithms, but you didn't give detail on the method/data you used.
Additionally, you might want to assess the shape of your PR-curve. An ideal PR-curve goes from the topleft corner horizontically to the topright corner and straight down to the bottomright corner, resulting in a PR-AUC of 1. In some applications, the PR-curve shows instead a strong spike at the beginning to quickly drop again close to the "random estimator line" (the horizontal line at 0.09 precision in your case). This would indicate a good detection of "strong" positive outcomes, but poor performance on the less clear candidates.
If you want to find a good threshold for your algorithm's cutoff parameter, you might consider the point on the PR-curve that's closest to the topright corner. Or even better, consider cross validation if possible. You might achieve precision and recall values for a specific cutoff parameter that are more interesting for your application than the value of the PR-AUC. The AUCs are most interesting when comparing different algorithms.
-
$\begingroup$ I would suspect the first sentence to be true,
would expect 0.09 for random classifier, but is there any documentation/research showing this? I find this topic sparse on the web. $\endgroup$ Feb 19, 2021 at 18:15 -
2$\begingroup$ See Saito & Rehmsmeier 2015 "The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets" :) $\endgroup$– EdgarAug 27, 2021 at 17:07
-
2$\begingroup$ I found this statement in that paper:
While the baseline is fixed with ROC, the baseline of PRC is determined by the ratio of positives (P) and negatives (N) as y = P / (P + N).Can find it at this link, confirming that the generality of PRC baselines is better than ROC. $\endgroup$ Aug 30, 2021 at 22:03
$\begingroup$
$\endgroup$
.49 is not great, but its interpretation is different than the ROC AUC. For ROC AUC, if you obtained a .49 using a logistic regression model, I would say you are doing no better than random. For .49 PR AUC, however it might not be that bad. I would consider looking at individual precision and recall, perhaps one or the other is what is driving down your PR AUC. Recall will tell you how much of that 9% positive class you are actually guessing correct. Precision will tell you how many you guessed positive that were not. (False Positives). 50% recall would be bad meaning you're not guessing many of your imbalanced class, but perhaps 50% precision wouldn't bad. Depends on your situation.
