12
$\begingroup$

A typical approach to solving a classification problem is to identify a class of candidate models, and then perform model selection using some procedure like cross validation. Typically one selects the model with the highest accuracy, or some related function that encodes problem specific information, like $\text{F}_\beta$.

Assuming the end goal is to produce an accurate classifier (where the definition of accuracy is again, problem dependent), in what situations is it better to perform model selection using a proper scoring rule as opposed to something improper, like accuracy, precision, recall, etc? Furthermore, let's ignore issues of model complexity and assume a priori we consider all the models equally likely.

Previously I would have said never. We know, in a formal sense, classification is an easier problem than regression [1], [2] and we can derive tighter bounds for the former than the later ($*$). Furthermore, there are cases when trying to accurately match probabilities can result in incorrect decision boundaries or overfitting. However, based on the conversation here and the voting pattern of the community in regards to such issues, I've been questioning this view.

  1. Devroye, Luc. A probabilistic theory of pattern recognition. Vol. 31. springer, 1996., Section 6.7
  2. Kearns, Michael J., and Robert E. Schapire. Efficient distribution-free learning of probabilistic concepts. Foundations of Computer Science, 1990. Proceedings., 31st Annual Symposium on. IEEE, 1990.

$(*)$ This statement might be a little sloppy. I specifically mean that given labeled data of the form $S = \{(x_1, y_1), \ldots, (x_n, y_n)\}$ with $x_i \in \mathcal{X}$ and $y_i \in \{1, \ldots, K\}$, it seems to be easier to estimate a decision boundary than accurately estimate conditional probabilities.

$\endgroup$

1 Answer 1

4
$\begingroup$

Think of this as a comparison between the $t$-test/Wilcoxon test and the Mood median test. The median test uses optimum classification (above or below the median for a continuous variable) so that it only loses $\frac{1}{\pi}$ of the information in the sample. Dichotomization at a point different from the median will lose much more information. Using an improper scoring rule such as proportion classified "correctly" is at most $\frac{2}{\pi}$ or about $\frac{2}{3}$ efficient. This results in selection of the wrong features and finding a model that is bogus.

$\endgroup$
19
  • 2
    $\begingroup$ The problem is that classification (as opposed to risk prediction) is an unnecessary dichotomization. $\endgroup$ Mar 25, 2014 at 2:32
  • 1
    $\begingroup$ I think I've explicitly assumed that issue away by stating I'm interested in classification. This is probably where the disagreement is coming from. We're solving two different problems. The goal of a properly posed classification problem is to produce an accurate classifier, not provide input for a decision maker. To give short shrift or outright dismiss the later as a useless endeavor is essentially dismissing the entire field (or at least a big chunk) of Machine Learning. $\endgroup$
    – alto
    Mar 25, 2014 at 16:15
  • 2
    $\begingroup$ There are numerous real world examples of problems that require making decisions. Normally these are things that are too difficult/boring/costly for a human to do. Things like spam classification, ad-serving, image recognition, content recommendation, speech recognition, etc. In some of these cases (spam classification, content recommendation), saying "I don't know" is the same as saying no. In other cases (speech recognition), the decision is forced. Something was said, and the algorithm needs to output something. $\endgroup$
    – alto
    Mar 25, 2014 at 18:42
  • 1
    $\begingroup$ Nice discussion. In some of the cases such with some spam detectors, you can get an 'uncertain'. I am more concerned with thresholding in problems such as medical diagnosis and prognosis. $\endgroup$ Mar 25, 2014 at 19:58
  • 2
    $\begingroup$ @FrankHarrell in your example, I think we are disagreeing / talking past each other on terminology. In the context of, say, covid triage " should patient get a ventilator? (Y/N)" , The option " see if patient improves on their own" is the 'no' decision ( aka not making a decision is also a decision). And ordering another diagnostic test is also a (third) decision. In the context I m applying this to, the decision making process is always necessarily immanent ( in particular triage situations). $\endgroup$ Mar 7, 2023 at 12:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.