Is it preferred to evaluate with a metric at a single decision threshold (eg Fbeta) vs averageing across thresholds (eg ROC-AUC)

Question

Consider these two approaches to evaluating a classifiers performance:

1. Choose a metric that summarizes the confusion matrix at a pre-determined decision threshold. Common suggestions seems to be Fbeta score for a model where True Negatives are not of great importance and Matthews correlation coefficient if True Negatives are of importance. There are of course many other metrics, but they all have in common that they are aggregating a confusion matrix derived from a pre-determined decision threshold (often p=0.5 by default in popular software such as sklearn).

2. Choose a metric that aggregates the performance over multiple decision. Commons suggestions seems to be to optimize AUC-ROC or average precision, which are both measures that don't relate to the performance at a single decision threshold, but rather integrates/averages the performance over multiple decision thresholds.

It seems to me that these two approaches are doing fundamentally different things, yet they are used interchangeably in many recommendations of metrics to choose when optimizing and/or comparing models. Approach 1 is finding the best possible model given that we have already decided on our decision threshold, and approach 2 is finding the model that has the most flexibility in adjusting the decision threshold while still getting good performance (ie it performs well at many thresholds). Q1. Is one of these two approaches theoretically more sound or preferable for other reasons (for example specific contexts when one is more helpful)?

Intuitively, it seems to me that approach one is unnecessarily inflexible and that approach two is not guaranteed to find the best performing model, just the most flexible one. Naively, I would think that a third approach would be appropriate where a combination of decision thresholds and context-appropriate confusion matrix aggregation metrics are optimized. Then the top performing models could be visualized in a way so that it is easy for the client to pick the trade-off between the different outcomes in the confusion matrix that they think is appropriate for their specific problem (e.g. instead of viewing a single model's PR curve as a line, we would view multiple combinations of models and decision thresholds plotted as points in PR space and pick the one that aligns the closest with the business objective). Q2 Would this be a reasonable approach or what are the shortcomings here? (I have seen this, but that does not quite answer what I'm asking here).

I'm have read some of the discussion around using proper scoring rules like Brier score and log loss to evaluate classifiers instead (such as this and this post), which seems helpful for both this and other issues, but it would still be helpful for me to understand the issues I asked about above.

Answer 1

It seems to me that these two approaches are doing fundamentally different things

They seem like they're different because they are!

When you calculate measures of performance such as log loss, Brier score, ROCAUC, or PRAUC, you are calculating the performance for the raw outputs of the model (arguably the ranks for the latter two). Notions of classification accuracy are nonexistent for such outputs.

When you calculate measures of performance such as accuracy, $F_{\beta}$, precision, recall, specificity, or Matthews' correlation, you are calculating the performance of a two-stage pipeline: first get the raw model outputs, then use some decision rule to map those raw outputs to categories. This second stage does not exist in the earlier situation.

If you truly need a classifier, then maybe you do need to tune and fix a decision rule to optimize some criterion (which might be accuracy or precision, preferably some measure of utility such as profit), in which case, you would evaluate the entire pipeline on that criterion. Harrell, however, argues in his Classification vs. Prediction blog post that such a requirement for strict classification is less common that it might appear, and his Damage Caused by Classification Accuracy and Other Discontinuous Improper Accuracy Scoring Rules blog post discusses related notions. A major argument of his is that the raw outputs allow you to have a grey zone where you might not want to take action. Kolassa discusses here a concrete example of what you can do with a grey zone (which was implemented on my work email at the job I held when he wrote that answer).

My MO in what I do is to assume a true classifier is not needed until someone can convince me otherwise. I largely side with Harrell that it is less common than most suspect. Even in automated systems where discrete decisions seem to be necessary, there can be a situation like Kolassa describes where there are more decisions than categories.