I have heard from multiple sources that a precision-recall curve is considered better than an ROC curve when testing a classifier on a dataset with a class imbalance. https://www.biostat.wisc.edu/~page/rocpr.pdf https://towardsdatascience.com/what-metrics-should-we-use-on-imbalanced-data-set-precision-recall-roc-e2e79252aeba

If this were the case I would expect that if I took an imbalanced dataset and resampled it to be balanced it would produce a significantly different ROC curve that the original dataset. However in my experience this is not the case according to my experiment or reason.

Can anyone explain to me how the ROC curve can be an okay measure of classification accuracy for balanced datasets, but not for unbalanced datasets, when the ROC curve seems to be essentially the same for both?


3 Answers 3


precision-recall curve is considered better than an ROC curve when testing a classifier on a dataset with a class imbalance.

This statement is plain wrong.

In general, statements like "X is better than Y" should be taken with a grain of salt. It usually depend on the use case, what is your target, etc. However, the statement above is more wrong than that. Let's take a look.

The PR curves plots the following parameters:

Precision = TP/(TP+FP)

Recall = TP/(TP+FN)

Notice how True Negatives (TN) are absent from the equation?

PR curves are useful when positive examples are rare. If your dataset is imbalanced with rare negatives, you should absolutely not use the PR curve.

As you noticed in your experiment, and as you have correctly reasoned, ROC curves are insensitive to class imbalance. This means that if you balance your data (ie with resampling), the ROC curve does not change (assuming you don't re-train your model).

When the dataset contains only few positive examples, you have a new problem to care about: positive predictive value (= PPV = Precision). Specifically: given an observation is classified as positive, what is the probability that it is really a True Positive? The answer to this question can be surprisingly misleading when positive examples are rare.

PPV and NPV (the complement: given an observation is negative, what is the probability to be a true negative) are usually not an issue in balanced datasets, as they follow the usual sensitivity and specificity. PPV and NPV only become critical in imbalanced datasets, because these two measures are sensitive to class imbalance, unlike ROC curves. So ROC curves can obscure models with poor PPV and NPV, which can be an issue in the case of imbalance. PR curves will immediately highlight models with poor PPV and totally disregard NPV.

So in the end it is up to you to choose which tool to use. Don't use PR curves just because you have imbalance. Are positive examples rare? Are they rare specifically in your sample? Then stick with ROC curves. Are they rare in the general population? Then you should consider precision and look at the PR curve too. Are negative cases rare? Then stick with ROC curve.

  • $\begingroup$ "This means that if you balance your data (ie with resampling), the ROC curve does not change." I don't think that's necessarily true, see e.g. arxiv.org/abs/1812.10857, esp. pages 7-9. $\endgroup$ Commented Jun 6, 2020 at 15:51
  • 1
    $\begingroup$ Balancing may of course affect model training, which is what this preprint is looking at. I was assuming a constant model, I tried to clarify my answer accordingly. $\endgroup$
    – Calimo
    Commented Jun 6, 2020 at 15:54

That's an interesting question. I don't agree with most people who say that ROC is always terrible for imbalanced data. It really depends on the use case and what you want to say with the ROC curve.

I assume why some people regard ROC as a bad metric for imbalanced problems could be that it focuses mostly on the positive examples and kind of neglects giving the same attention to the other (negative) class.

I.e., in ROC curves, you have TP/P and FP/N.

With Precision-Recall curves, you also focus on the correctly predicted negative examples via recall TN/N (i.e,. the true negatives, TN, are what make the difference).

In either case, I wouldn't say that PRE-REC are always to be preferred over ROC curves, it really depends on the context.


It is not always true.

PR curve considered better than an ROC curve for imbalanced datasets

It depends on the business problem and performance we need to measure.

ROC looks at TPR and FPR. Hence, looking at ROC only, we can miss FP > TP situation which is low precision. If low precision is not good for our problem, then using ROC only is not a good idea.

We need to ask:

  1. Will FP > TP happen with our data and the model.
  2. Does FP > TP matter for our problem.

If the answer is yes for both, then looking at ROC only is not fit for use to measure the performance. Looking at PR can help identify the low precision.

enter image description here

FP: False Positive
TP: True Positive
FPR: False Positive Rate
TPR: True Positive Rate

Suppose I am running a risky loan business where many customers can be risky. If I evaluate the business performance with ROC only, I may think the performance is good although actually the business is losing money because of FP > TP. Then measuring the performance only with ROC is not fit for the purpose.

enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.