Skip to main content
3 of 3
added 8 characters in body
Dave
  • 67.1k
  • 7
  • 105
  • 305

Do you think this is a good criterion for selecting a classification algorithm?

While area under the receiver-operator characteristic curve has an interpretation as how well the predictions for the two categories are separated (so a clear interpretation as a measure of model ability to distinguish between the two categories), the interpretation of the area under the PR curve as the average precision across all thresholds is not satisfying for me.

I actually like PR curves as a visualization, perhaps more than ROC curves, as they give a sense of the tradeoff between the ability to detect positive cases vs the probability of crying wolf, which is probably a real concern. (You want the person looking after the sheep to see when there is a wolf on the loose, but you also want to be alerted about a wolf when there is a high probability that there really is a wolf.) The sensitivity and specificity in a ROC curve get at this, sure, but it is made explicit in the PR curve, since the relationship between precision, recall/sensitivity, and specificity is not super straightforward.

Thus, I lack an intuition for the area under the PR curve and why I should care about that value, even if I find the curve itself to be a useful visualization. Yes, a higher number is better, but it is not always clear what constitutes a high number, since the baseline average precision depends on the data. With a ROC curve the baseline performance is $0.5$ every time, so we can assess that a ROCAUC of $0.55$ is a small improvement over the baseline while $0.9$ is a large improvement. With PR curves, it could be that a PRAUC of $0.9$ is worse than baseline in some cases (if the prevalence is below $10\%$) while a lower PRAUC of $0.7$ could be an improvement over the baseline (if the prevalence is above $30\%$). Thus, area under the PR curve does not even tell the whole story about performance in terms of the tradeoff between precision and recall, and it requires comparison to a baseline that changes for every problem (but is always $0.5$ for ROCAUC).

As a bit of a side comment, if you really are doing classification, you need to have some rule to bin your continuous predictions into discrete categories, typically a rule that predictions below a threshold value are categorized as negative cases and predictions above a threshold are categorized as positive cases. Once you have this, you do not care how other categorization rules perform. You only care about the performance with this particular threshold (since that is the full decision pipeline), not across all thresholds.

Are recall and PR_AUC proportional?

These metrics do not apply to the same model. The area under the PR curve applies to the original model that makes predictions on a continuum, such as a logistic regression or neural network. Recall applies to the pipeline that starts with such a model but also applies a threshold (or maybe some more complicated classification rule) to convert the continuous predictions into discrete categories. PR curves do not even correspond to one recall value. Thus, this question turns out not to make sense, even though it seems like it should. Every PR curve and the area underneath it corresponds with a range of recall values, some of which will be good and some of which will be bad (depending on the threshold, recall values for any model can span from zero to perfect, as I believe I demonstrate here).

Which one is the best criterion for imbalanced datasets?

Class imbalance is much less of a problem than it sometimes seems when proper statistical methods are applied. My recommendation is to use the classics of log loss and/or Brier score, perhaps scaling them with $R^2$-style calculations to ease the interpretation. The linked equations might look intimidating, but they turn out to make a lot of sense, and software handles the nasty calculations.

I will close by linking some helpful resources related to class imbalance and classification problem performance evaluation.

Are unbalanced datasets problematic, and (how) does oversampling (purport to) help?

Profusion of threads on imbalanced data - can we merge/deem canonical any?

Why is accuracy not the best measure for assessing classification models?

Academic reference on the drawbacks of accuracy, F1 score, sensitivity and/or specificity

Proper scoring rule when there is a decision to make (e.g. spam vs ham email)

Harrell’s Blog: Classification vs. Prediction

Harrell’s Blog: Damage Caused by Classification Accuracy and Other Discontinuous Improper Accuracy Scoring Rules

(Note that “recall” and “sensitivity” are synonyms when it comes to classification problem performance metrics in statistics and machine learning.)

Dave
  • 67.1k
  • 7
  • 105
  • 305