22
$\begingroup$

Accuracy, as a KPI for assessing binary classification models, has major drawbacks: Why is accuracy not the best measure for assessing classification models?. The exact same issues also plague the F1 score (actually all Fβ scores), sensitivity, specificity and alternatives.

Is there a standard academic article one can point to discussing these issues?

Why am I asking this? I am thinking of reviewing a paper and wanting the author to avoid these KPIs. Or alternatively, having submitted a paper, getting reviews that recommend these flawed KPIs, and needing a paper to point to in arguing why I won't follow these recommendations. Of course, I could point to the CV thread linked above, but unfortunately, CV is not always accorded the respect a peer-reviewed article gets.

I have looked through Frank Harrell's "Damage Caused by Classification Accuracy and Other Discontinuous Improper Accuracy Scoring Rules". This kind of material is exactly what I am envisaging. Is there something like this published somewhere?

$\endgroup$
35
  • 5
    $\begingroup$ @Him Accuracy, recall, precision, confusion matrices, and $F_1$ ($F_{\beta}$) are exactly the types of performance metrics that Stephan wants to avoid. $\endgroup$
    – Dave
    Commented Jan 31, 2023 at 19:13
  • 3
    $\begingroup$ @Him What about log loss or Brier score? Those are the types of metrics for which Stephan wants to argue (though I admit that is not entirely clear from just the OP). $\endgroup$
    – Dave
    Commented Jan 31, 2023 at 19:18
  • 5
    $\begingroup$ @Him: sorry, but your last comment is simply wrong. The Brier score is a function of probabilistic classifications, and the confusion matrix is (can be) a function of these probabilistic classifications plus a threshold. Thus, the Brier (or log) score is not just a function of the confusion matrix, or of accuracy/sensitivity/specificity. (A recurring theme in my answers here is that using a default threshold (like 0.5) is often a terrible idea.) $\endgroup$ Commented Jan 31, 2023 at 19:26
  • 4
    $\begingroup$ @Him: precisely. The $f_{ti}$ are probabilities, accuracy etc. deal with hard 0-1 classifications. I agree that the underlying problem is that of hard classifications via thresholding. The problem is that people do not see that this is the underlying problem. Contrary to your earlier comments, I believe that most ML textbooks do not teach any of this, and solely discuss accuracy and friends. At least that is the impression I get from the almost daily questions here on CV that reveal zero understanding of this issue. $\endgroup$ Commented Jan 31, 2023 at 19:33
  • 3
    $\begingroup$ @Dave my issue was that "Surely this issue is brought up in numerous textbooks on the subject of Machine Learning." However, upon looking into the matter in several (several) introductory and intermediate-level textbooks on machine learning, I saw that, in fact, most of them don't discuss how to measure the performance of a model at all. Often, they simply start measuring a thing, and the reader is expected to just assume that this metric is a performance metric of some sort. This is utterly astonishing to me, but nevertheless seems to be the case. $\endgroup$
    – Him
    Commented Feb 2, 2023 at 2:05

2 Answers 2

18
+100
$\begingroup$

The main one that springs to mind is "Three myths about risk thresholds for prediction models" by Wynants et al. (2019) where they argue strongly against using a "universally optimal threshold" without context. I liked they used the term "dichotomania" too (in effect meaning: "manically dichotomising continuous variables").

I like Peter Flach's work on the area of "evaluating ML model performance" too. I do not have a single definitive reference there but something like Berrar's and his: "Caveats and pitfalls of ROC analysis in clinical microarray research (and how to avoid them)" (2012) is a reasonable point to start. His "Precision-recall-gain curves: PR analysis done right" (2015) with Kull has been very thought-provoking too.

$\endgroup$
1
  • 1
    $\begingroup$ +1 Even the abstract of the Wynants article is quite biting. $\endgroup$
    – Dave
    Commented Jan 30, 2023 at 18:19
1
$\begingroup$

The KPI (Key Performance Indicator) depends on the requirements of the application. For some applications (i.e. those where a hard classification must be made and we know a-priori that the misclassification costs are equal, e.g. some handwritten character recognition tasks) accuracy is a completely reasonable performance metric and it would be a mistake to recommend avoiding it because it has problems as well as advantages.

Similarly, for some applications (primarily information retrieval) where it is more natural to talk of the relative importance of precision and recall than of misclassification costs, then $F_1$ or more generally $F_\beta$ may be appropriate, especially where we need to make a decision ("do I read this article, or don't I?").

An important consideration is whether we need to make a decision. We may well implement the system using a probabilistic classifier, and then applying a threshold. However, if we need a decision, then the performance of the system depends on the setting of that threshold, so we should be using a performance metric that depends on the threshold, as we need to include the effects of the threshold on the performance of the system.

The advice I would give is not to have a single KPI, but have a range of performance metrics that provide information on different aspects of classifier performance. I quite often use accuracy (to measure the quality of the decisions), or equivalently the expected risk where misclassification costs are unequal, the area under the receiver operating characteristic (to measure the ranking of samples) and the cross-entropy (or similar) to measure the calibration of the probability estimates.

Basically, our job as statisticians is to understand the advantages and disadvantages of performance metrics so that we can select the appropriate metric(s) for the needs of the application. All metrics have advantages and disadvantages, and we shouldn't reject any of them a-priori because of their disadvantages if they have advantages or relevance for our application. I think the advantages and disadvantages are well covered in textbooks (even ML ones! ;o), so I would just use those.

Also, as I have said elsewhere, we should make a distinction between performance estimation and model selection. They are not the same problem, and sometimes we should have different metrics for each task.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.