2
$\begingroup$

I’ve read that precision-recall (PR) curves are preferred over AUC-ROC curves when a dataset is imbalanced as there’s more of a focus on the model’s performance in correctly identifying the minority/positive class.

At what point (rule of thumb?) does it make more sense to primarily use PR to evaluate a classifier instead of AUC-ROC score? I imagine if the dataset has 40% positive class, AUC is still appropriate? But what about at 30% or 20% positive class? What level is considered “imbalanced” where PR is preferred?

Improve this question
$\endgroup$
5
  • 2
    $\begingroup$ "Unbalanced" datasets are not a problem: Are unbalanced datasets problematic, and (how) does oversampling (purport to) help? However, precision and recall are: Why is accuracy not the best measure for assessing classification models? (everything said about accuracy at that thread also applies to precision and recall). $\endgroup$
    – Stephan Kolassa
    May 11, 2020 at 4:42
  • $\begingroup$ @StephanKolassa so what’s the rule of thumb? I read the links and most of the examples were 1% positive class and 99% negative class. Are you suggesting that’s the answer? $\endgroup$
    – Insu Q
    May 11, 2020 at 12:31
  • 1
    $\begingroup$ No. Per my question and my answer to the accuracy question, there is no problem with unbalanced data, unless you use inappropriate quality measures like accuracy. Use an appropriate probabilistic model, and "unbalance" will naturally be expressed as low probabilities. $\endgroup$
    – Stephan Kolassa
    May 11, 2020 at 14:20
  • $\begingroup$ @StephanKolassa I might not have asked my question correctly. I know there’s no problem with unbalanced data. A lot of real-world data is unbalanced. My question is, is there a point in that level of unbalance where using PR curves makes more sense than using AUC? If you have too few positive examples in a dataset, the AUC can appear to be high and when you look at the PR curve, it’s obvious there’s room for improvement. When your dataset has 49% positives and 51% negatives, technically it’s unbalanced but AUC is fine to use. When it’s 5% positives, you probably want to look at a PR curve. $\endgroup$
    – Insu Q
    May 11, 2020 at 14:30
  • 1
    $\begingroup$ I advocate not using precision/recall at all. See the links above for my argument. This may be helpful for context. $\endgroup$
    – Stephan Kolassa
    May 11, 2020 at 14:43

2 Answers 2

Reset to default
1
$\begingroup$

Context

The imbalance depends on the dataset size also.

A model with 5-10% positive class and 90-95% negative class with 50 or 500 samples is different from a model that has 10'000 samples.

Opinion

A model seeing 1 positive sample and trying to learn from it is different from seeing hundreds of positive samples (even if they represent only 5% of the whole data).

Anyway, as anything between 20-40% positives is considered imbalanced, too imbalanced is around 5-10%, and extremely imbalanced is below 5%.

Resampling

Multiple resampling methods exist, however, it is very tricky on whether or not they improve your model, since an increase in the recall, causes also a huge decrease in precision in most of the times (if you oversample the minority).

Improve this answer
$\endgroup$
5
  • $\begingroup$ Consideration of imbalance means that you probability don't have a proper accuracy scoring rule in your mind. Take a look at fharrell.com/post/class-damage $\endgroup$
    – Frank Harrell
    Nov 24, 2020 at 12:45
  • 1
    $\begingroup$ @FrankHarrell please provide an answer to the post. $\endgroup$
    – ombk
    Nov 24, 2020 at 12:47
  • $\begingroup$ I posted my comments as a comment rather than an answer. $\endgroup$
    – Frank Harrell
    Nov 24, 2020 at 13:21
  • $\begingroup$ @FrankHarrell what if our data is not linearly separable and we are just using a basic model like logistic regression. $\endgroup$
    – ombk
    Nov 24, 2020 at 13:24
  • $\begingroup$ Not clear on what that means. Extremely easy to relax linearity assumptions - see my RMS course notes. $\endgroup$
    – Frank Harrell
    Nov 24, 2020 at 21:08
0
$\begingroup$

Agree with the comments, I have used AUC ROC for binary classification with a class imbalance of 5% positive and 95% negative. I was actually able to get a pretty good model still.

Improve this answer
$\endgroup$
1
  • $\begingroup$ The concordance probability (AUROC) is not used for classification (forced choice) but rather for assessing the pure predictive discrimination of a continuous prediction. And as you said it is unaffected by extreme imbalance. $\endgroup$
    – Frank Harrell
    Nov 24, 2020 at 12:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.