Consider these two approaches to evaluating a classifiers performance:

  1. Choose a metric that summarizes the confusion matrix at a pre-determined decision threshold. Common suggestions seems to be Fbeta score for a model where True Negatives are not of great importance and Matthews correlation coefficient if True Negatives are of importance. There are of course many other metrics, but they all have in common that they are aggregating a confusion matrix derived from a pre-determined decision threshold (often p=0.5 by default in popular software such as sklearn).

  2. Choose a metric that aggregates the performance over multiple decision. Commons suggestions seems to be to optimize AUC-ROC or average precision, which are both measures that don't relate to the performance at a single decision threshold, but rather integrates/averages the performance over multiple decision thresholds.

It seems to me that these two approaches are doing fundamentally different things, yet they are used interchangeably in many recommendations of metrics to choose when optimizing and/or comparing models. Approach 1 is finding the best possible model given that we have already decided on our decision threshold, and approach 2 is finding the model that has the most flexibility in adjusting the decision threshold while still getting good performance (ie it performs well at many thresholds). Q1. Is one of these two approaches theoretically more sound or preferable for other reasons (for example specific contexts when one is more helpful)?

Intuitively, it seems to me that approach one is unnecessarily inflexible and that approach two is not guaranteed to find the best performing model, just the most flexible one. Naively, I would think that a third approach would be appropriate where a combination of decision thresholds and context-appropriate confusion matrix aggregation metrics are optimized. Then the top performing models could be visualized in a way so that it is easy for the client to pick the trade-off between the different outcomes in the confusion matrix that they think is appropriate for their specific problem (e.g. instead of viewing a single model's PR curve as a line, we would view multiple combinations of models and decision thresholds plotted as points in PR space and pick the one that aligns the closest with the business objective). Q2 Would this be a reasonable approach or what are the shortcomings here? (I have seen this, but that does not quite answer what I'm asking here).

I'm have read some of the discussion around using proper scoring rules like Brier score and log loss to evaluate classifiers instead (such as this and this post), which seems helpful for both this and other issues, but it would still be helpful for me to understand the issues I asked about above.


1 Answer 1


It seems to me that these two approaches are doing fundamentally different things

They seem like they're different because they are!

When you calculate measures of performance such as log loss, Brier score, ROCAUC, or PRAUC, you are calculating the performance for the raw outputs of the model (arguably the ranks for the latter two). Notions of classification accuracy are nonexistent for such outputs.

When you calculate measures of performance such as accuracy, $F_{\beta}$, precision, recall, specificity, or Matthews' correlation, you are calculating the performance of a two-stage pipeline: first get the raw model outputs, then use some decision rule to map those raw outputs to categories. This second stage does not exist in the earlier situation.

If you truly need a classifier, then maybe you do need to tune and fix a decision rule to optimize some criterion (which might be accuracy or precision, preferably some measure of utility such as profit), in which case, you would evaluate the entire pipeline on that criterion. Harrell, however, argues in his Classification vs. Prediction blog post that such a requirement for strict classification is less common that it might appear, and his Damage Caused by Classification Accuracy and Other Discontinuous Improper Accuracy Scoring Rules blog post discusses related notions. A major argument of his is that the raw outputs allow you to have a grey zone where you might not want to take action. Kolassa discusses here a concrete example of what you can do with a grey zone (which was implemented on my work email at the job I held when he wrote that answer).

My MO in what I do is to assume a true classifier is not needed until someone can convince me otherwise. I largely side with Harrell that it is less common than most suspect. Even in automated systems where discrete decisions seem to be necessary, there can be a situation like Kolassa describes where there are more decisions than categories.

  • $\begingroup$ Thank you for the quick and detailed reply, it is really helpful! I had not realized that the ROC-AUC and PR-AUC (and AP?) could be seen a rank based metrics. I agree that having a grey zone of probabilities/ranks that treated differently is an appealing reason to use Brier scores or similar metrics. If one does not go the route of Brier score and the likes, it seems to me that the third approach I suggested (optimizing the decision threshold as a hyperparameter during CV) would still have some value, would you agree with this or am I overstating its potential utility? $\endgroup$ Nov 18, 2023 at 3:22
  • $\begingroup$ A similar question here, but the answer lacks detail. $\endgroup$ Nov 18, 2023 at 3:22
  • $\begingroup$ @another_student I'd really encourage you to read the material from Harrell and Kolassa about what these models do and what utility you get from modeling the probabilities. I do see value to looking at what various classification metrics are across multiple thresholds, but that is really more about getting a sense of how good the model is at separating the categories, not an assessment of how well the probabilities are predicted. You don't necessarily have to threshold at all, and there are often considerable advantages to avoiding such thresholding. $\endgroup$
    – Dave
    Nov 20, 2023 at 14:42
  • $\begingroup$ Great, thank you for the reply @Dave! I have been reading a fair bit of the answers from Harrell and Kolassa here on CV and they have been insightful, and your answer here was very helpful in clarifying some concepts for me that wasn't explicit in what I have read so far. I will try to dive deeper into some of their articles and blog posts as well to understand this further. Thanks again. $\endgroup$ Nov 21, 2023 at 19:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.