I'm trying to evaluate some classification algorithms' results in my imbalanced dataset. By imbalanced, I mean there are much more negative than positive labels. Accuracy and precision are always good, but recall, and Area Under the Precision-Recall curve (PR_AUC) are not so good. I'm seeking the classifier that maximizes the PR_AUC.

1.- Do you think this is a good criterion for selecting a classification algorithm?

2.- Are recall and PR_AUC proportional? I mean, if a classifier gives better recall results than another one but worse PR_AUC results... Am I doing something wrong? Or it has a logical explanation? Which one is the best criterion for imbalanced datasets?

  • $\begingroup$ "Do you think this is a good criterion for selecting a classification algorithm?" this rather depends on what you are trying to achieve, which depends on the requirements of the application. More information is needed to give a satisfactory answer. Why are recall and precision good ways of viewing performance for your application? For most of the applications I look at, it is more natural to think about unequal misclassification costs rather than precision/recall. Note the performance metric depends on the nature of the application, not the qualities of the dataset (such as imbalance). $\endgroup$ Commented Jun 5, 2023 at 16:22

2 Answers 2


Do you think this is a good criterion for selecting a classification algorithm?

While area under the receiver-operator characteristic curve has an interpretation as how well the predictions for the two categories are separated (so a clear interpretation as a measure of model ability to distinguish between the two categories), the interpretation of the area under the PR curve as the average precision across all thresholds is not satisfying for me.

I actually like PR curves as a visualization, perhaps more than ROC curves, as they give a sense of the tradeoff between the ability to detect positive cases vs the probability of crying wolf, which is probably a real concern. (You want the person looking after the sheep to see when there is a wolf on the loose, but you also want to be alerted about a wolf when there is a high probability that there really is a wolf.) The sensitivity and specificity in a ROC curve get at this, sure, but it is made explicit in the PR curve, since the relationship between precision, recall/sensitivity, and specificity is not super straightforward.

Thus, I lack an intuition for the area under the PR curve and why I should care about that value, even if I find the curve itself to be a useful visualization. Yes, a higher number is better, but it is not always clear what constitutes a high number, since the baseline average precision depends on the data. With a ROC curve the baseline performance is $0.5$ every time, so we can assess that a ROCAUC of $0.55$ is a small improvement over the baseline while $0.9$ is a large improvement. With PR curves, it could be that a PRAUC of $0.9$ is worse than baseline in some cases (if the prevalence is below $10\%$) while a lower PRAUC of $0.7$ could be an improvement over the baseline (if the prevalence is above $30\%$). Thus, area under the PR curve does not even tell the whole story about performance in terms of the tradeoff between precision and recall, and it requires comparison to a baseline that changes for every problem (but is always $0.5$ for ROCAUC).

As a bit of a side comment, if you really are doing classification, you need to have some rule to bin your continuous predictions into discrete categories, typically a rule that predictions below a threshold value are categorized as negative cases and predictions above a threshold are categorized as positive cases. Once you have this, you do not care how other categorization rules perform. You only care about the performance with this particular threshold (since that is the full decision pipeline), not across all thresholds.

Are recall and PR_AUC proportional?

These metrics do not apply to the same model. The area under the PR curve applies to the original model that makes predictions on a continuum, such as a logistic regression or neural network. Recall applies to the pipeline that starts with such a model but also applies a threshold (or maybe some more complicated classification rule) to convert the continuous predictions into discrete categories. PR curves do not even correspond to one recall value. Thus, this question turns out not to make sense, even though it seems like it should. Every PR curve and the area underneath it corresponds with a range of recall values, some of which will be good and some of which will be bad (depending on the threshold, recall values for any model can span from zero to perfect, as I believe I demonstrate here).

Which one is the best criterion for imbalanced datasets?

Class imbalance is much less of a problem than it sometimes seems when proper statistical methods are applied. My recommendation is to use the classics of log loss and/or Brier score, perhaps scaling them with $R^2$-style calculations to ease the interpretation. The linked equations might look intimidating, but they turn out to make a lot of sense, and software handles the nasty calculations.

I will close by linking some helpful resources related to class imbalance and classification problem performance evaluation.

Are unbalanced datasets problematic, and (how) does oversampling (purport to) help?

Profusion of threads on imbalanced data - can we merge/deem canonical any?

Why is accuracy not the best measure for assessing classification models?

Academic reference on the drawbacks of accuracy, F1 score, sensitivity and/or specificity

Proper scoring rule when there is a decision to make (e.g. spam vs ham email)

Harrell’s Blog: Classification vs. Prediction

Harrell’s Blog: Damage Caused by Classification Accuracy and Other Discontinuous Improper Accuracy Scoring Rules

(Note that “recall” and “sensitivity” are synonyms when it comes to classification problem performance metrics in statistics and machine learning.)


Have a look at the paper "The Relationship Between Precision-Recall and ROC Curves" by Jesse Davis and Mark Goadrich

"[... a classifier] dominates in ROC space if and only if it dominates in PR space"

  • $\begingroup$ This does not answer the question; The OP mentioned the relation between the recall and AUC in PR space. The paper you mentioned is about the relationship between ROC space and PR space $\endgroup$
    – James
    Commented May 19, 2023 at 13:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.