Why is a PR curve considered better than an ROC curve for imbalanced datasets?

Question

I have heard from multiple sources that a precision-recall curve is considered better than an ROC curve when testing a classifier on a dataset with a class imbalance. https://www.biostat.wisc.edu/~page/rocpr.pdf https://towardsdatascience.com/what-metrics-should-we-use-on-imbalanced-data-set-precision-recall-roc-e2e79252aeba

If this were the case I would expect that if I took an imbalanced dataset and resampled it to be balanced it would produce a significantly different ROC curve that the original dataset. However in my experience this is not the case according to my experiment or reason.

Can anyone explain to me how the ROC curve can be an okay measure of classification accuracy for balanced datasets, but not for unbalanced datasets, when the ROC curve seems to be essentially the same for both?

Answer 1

precision-recall curve is considered better than an ROC curve when testing a classifier on a dataset with a class imbalance.

This statement is wrong.

In general, statements like "X is better than Y" should be taken with a grain of salt. It usually depend on the use case, what is your target, etc. However, the statement above is more wrong than that. Let's take a look.

The PR curves plots the following parameters:

Precision = TP/(TP+FP)

Recall = TP/(TP+FN)

Notice how True Negatives (TN) are absent from the equation?

PR curves are useful when positive examples are rare. If your dataset is imbalanced with rare negatives, you should absolutely not use the PR curve.

---

As you noticed in your experiment, and as you have correctly reasoned, ROC curves are insensitive to class imbalance. This means that if you balance your data (ie with resampling), the ROC curve does not change.

When the dataset contains only few positive examples, you have a new problem to care about: positive predictive value (= PPV = Precision). Specifically: given an observation is classified as positive, what is the probability that it is really a True Positive? The answer to this question can be surprisingly misleading when positive examples are rare.

PPV and NPV (the complement: given an observation is negative, what is the probability to be a true negative) are usually not an issue in balanced datasets, as they follow the usual sensitivity and specificity. PPV and NPV only become critical in imbalanced datasets, because these two measures are sensitive to class imbalance, unlike ROC curves. So ROC curves can obscure models with poor PPV and NPV, which can be an issue in the case of imbalance. PR curves will immediately highlight models with poor PPV and totally disregard NPV.

So in the end it is up to you to choose which tool to use. Don't use PR curves just because you have imbalance. Are positive examples rare? Are they rare specifically in your sample? Then stick with ROC curves. Are they rare in the general population? Then you should consider precision and look at the PR curve too. Are negative cases rare? Then stick with ROC curve.

Answer 2

That's an interesting question. I don't agree with most people who say that ROC is always terrible for imbalanced data. It really depends on the use case and what you want to say with the ROC curve.

I assume why some people regard ROC as a bad metric for imbalanced problems could be that it focuses mostly on the positive examples and kind of neglects giving the same attention to the other (negative) class.

I.e., in ROC curves, you have TP/P and FP/N.

With Precision-Recall curves, you also focus on the correctly predicted negative examples via recall TN/N (i.e,. the true negatives, TN, are what make the difference).

In either case, I wouldn't say that PRE-REC are always to be preferred over ROC curves, it really depends on the context.