11
$\begingroup$

I'm working with imbalanced data, where there are about 40 class=0 cases for every class=1. I can reasonably discriminate between the classes using individual features, and training a naive Bayes and SVM classifier on 6 features and balanced data yielded better discrimination (ROC curves below).

enter image description here

That's fine, and I thought I was doing well. However, the convention for this particular problem is to predict hits at a precision level, usually between 50% and 90%. e.g. "We detected some-number of hits at 90% precision." When I tried this, the maximum precision I could get from the classifiers was about 25% (black line, PR curve below).

I could understand this as a class imbalance problem, since PR curves are sensitive to imbalance and ROC curves aren't. However, the imbalance doesn't seem to affect the individual features: I can get pretty high precision using the individual features (blue and cyan).

enter image description here

I don't understand what's going on. I could understand it if everything performed badly in PR space, since, after all, the data is very imbalanced. I could also understand it if the classifiers looked bad in ROC and PR space - maybe they're just bad classifiers. But what is going on to make the classifiers better as judged by ROC, but worse as judged by Precision-Recall?

Edit: I noticed that in the low TPR/Recall areas (TPR between 0 and 0.35), the individual features consistently outperform the classifiers in both ROC and PR curves. Maybe my confusion is because the ROC curve "emphasizes" the high TPR areas (where the classifiers do well) and the PR curve emphasizes the low TPR (where the classifiers are worse).

Edit 2: Training on nonbalanced data, i.e. with the same imbalance as the raw data, brought the PR curve back to life (see below). I'd guess my problem was improperly training the classifiers, but I don't totally understand what happened. enter image description here

Improve this question
$\endgroup$

3 Answers 3

Reset to default
8
$\begingroup$

I've found that there isn't an incredible benefit in using downsampling/upsampling when classes are moderately imbalanced (i.e., no worse than 100:1) in conjunction with a threshold invariant metric (like AUC). Sampling makes the biggest impact for metrics like F1-score and Accuracy, because the sampling artificially moves the threshold to be closer to what might be considered as the "optimal" location on an ROC curve. You can see an example of this in the caret documentation.

I would disagree with @Chris in that having a good AUC is better than precision, as it totally relates to the context of the problem. Additionally, having a good AUC doesn't necessarily translate to a good Precision-Recall curve when the classes are imbalanced. If a model shows good AUC, but still has poor early retrieval, the Precision-Recall curve will leave a lot to be desired. You can see a great example of this happening in this answer to a similar question. For this reason, Saito et al. recommend using area under the Precision-Recall curve rather than AUC when you have imbalanced classes.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ I didn't intend to imply that a good AUC is always better than a good PR curve. $\endgroup$
    – Chris
    Mar 16, 2016 at 5:02
  • 1
    $\begingroup$ Note also that you can always flip the positive and negative labels before fitting the model and maximizing AUC-PR. The popular document-retrieval example exhibits the fact that people usually like the AUC-PR because it can be used to minimize false hits; they're clearly bothered more by unrelated documents they're forced to look at than by relevant documents that they miss. I study wars... so, to put it lightly, I'm much more bothered by missed hits than false alarms. But that means I just use AUC-PR with peace as the positive. I'd only use ROC if I had no preference regarding error type. $\endgroup$
    – DHW
    Jul 19, 2018 at 18:32
2
$\begingroup$

I could understand this as a class imbalance problem, since PR curves are sensitive to imbalance and ROC curves aren't. However, the imbalance doesn't seem to affect the individual features: I can get pretty high precision using the individual features (blue and cyan).

May I just point out that this is actually the other way around: ROC is sensitive to class imbalance while PR is more robust when dealing with skewed class distributions. See https://www.biostat.wisc.edu/~page/rocpr.pdf.

They also show that "algorithms that optimize the area under the ROC curve are not guaranteed to optimize the area under the PR curve."

Improve this answer
$\endgroup$
1
$\begingroup$

The best way to evaluate a model is to look at how it will be used in the real world and develop a cost function.

As an aside, for example, there is too much emphasis on r squared but many believe it is a useless statistic. So do not get hung up on any one statistic.

I suspect that your answer is an example of the accuracy paradox.

https://en.m.wikipedia.org/wiki/Accuracy_paradox

Recall (also known as sensitivity aka true positive rate) is the fraction of relevant instances that are retrieved.

tpr = tp / ( tp + fn )

Precision (aka positive predictive value) is the fraction of retrieved instances that are relevant.

ppv = tp / (tp + fp)

Let's say you have a very imbalanced set of 99 positives and one negative.

Let's say a model is trained in which the model says everything is positive.

tp = 99 fp = 1 ppv becomes 0.99

Clearly a junk model despite the "good" positive predictive value.

I recommend building a training set that is more balanced either through oversampling or undersampling. After the model is built then use a validation set that the keeps the original imbalance and build a performance chart on that.

Improve this answer
$\endgroup$
9
  • $\begingroup$ Thanks so much. I didn't mention it, but I'm training on balanced data (done with re-sampling) and evaluating on the imbalanced data. Re: accuracy paradox, do you mean that the classifier might just be picking the dominant class? Wouldn't that lead to a random-looking ROC curve? The "good" classifier in your example has a good Precision and a bad ROC; the "good" classifier in my case has the opposite, a bad Precision but a good ROC. $\endgroup$
    – R Greg Stacey
    Mar 15, 2016 at 19:10
  • $\begingroup$ Omitting valuable data is not the correct solution. This is a wasteful way of dealing with a problem that comes from using improper accuracy scoring rules. In addition, the proposed strategy laid out in the original question is at odds with optimal decision making. $\endgroup$
    – Frank Harrell
    Mar 15, 2016 at 20:33
  • $\begingroup$ Where does he state that he is omitting valuable data? $\endgroup$
    – Chris
    Mar 15, 2016 at 20:37
  • 1
    $\begingroup$ @Frank Thanks for the comment. What do you mean by "using improper accuracy scoring rules"? Also, which "proposed strategy is at odds with optimal decision making"? Do you mean defining hits at a given precision level, or something else? $\endgroup$
    – R Greg Stacey
    Mar 15, 2016 at 20:39
  • 1
    $\begingroup$ It's not enough to evaluate a model within the business case. The business case should be solved optimally by building an optimal model and apply a utility function on top of that for making decisions, not by building any thresholds into the analysis. Discontinuous accuracy scores and other improper accuracy scores allow one to claim that a bogus (wrong) model is superior, and leads one to select the wrong features and other harms when the improper accuracy score is used to build the model. The fact that @Qroid discarded data is great evidence for the accuracy assessment being faulty. $\endgroup$
    – Frank Harrell
    Mar 15, 2016 at 22:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.