62
$\begingroup$

I have a question regarding classification in general. Let $f$ be a classifier, which outputs a set of probabilities given some data D. Normally, one would say: well, if $P(c|D) > 0.5$, we will assign a class 1, otherwise 0 (let this be a binary classification).

My question is, what if I find out, that if I classify the class as 1 also when the probabilities are larger than, for instance 0.2, and the classifier performs better. Is it legitimate to then use this new threshold when doing classification?

I would interpret the necessity for lower classification bound in the context of the data emitting a smaller signal; yet still significant for the classification problem.

I realize this is one way to do it. However, if this is not correct thinking of reducing the threshold, what would be some data transformations, which emphasize individual features in a similar manner, so that the threshold can remain at 0.5?

Improve this question
$\endgroup$
  • 11
    $\begingroup$ You already have some excellent answer, so let me just say this: your "normally" is not a normally that should be normal. I'm not sure where the "threshold at 0.5" thing became standard, and I know there is some, otherwise excellent, software that encourages the idea, but it's a very poor practice in general. $\endgroup$ – Matthew Drury Nov 6 '17 at 15:25
  • 2
    $\begingroup$ @MatthewDrury: unless, of course, the score is the well-calibrated relevant posterior probability of making no important mistake (the latter would take care of different costs of misclassification). $\endgroup$ – cbeleites unhappy with SX Nov 6 '17 at 21:43
91
$\begingroup$

Frank Harrell has written about this on his blog: Classification vs. Prediction, which I agree with wholeheartedly.

Essentially, his argument is that the statistical component of your exercise ends when you output a probability for each class of your new sample. Choosing a threshold beyond which you classify a new observation as 1 vs. 0 is not part of the statistics any more. It is part of the decision component. And here, you need the probabilistic output of your model - but also considerations like:

  • What are the consequences of deciding to treat a new observation as class 1 vs. 0? Do I then send out a cheap marketing mail to all 1s? Or do I apply an invasive cancer treatment with big side effects?
  • What are the consequences of treating a "true" 0 as 1, and vice versa? Will I tick off a customer? Subject someone to unnecessary medical treatment?
  • Are my "classes" truly discrete? Or is there actually a continuum (e.g., blood pressure), where clinical thresholds are in reality just cognitive shortcuts? If so, how far beyond a threshold is the case I'm "classifying" right now?
  • Or does a low-but-positive probability to be class 1 actually mean "get more data", "run another test"?

So, to answer your question: talk to the end consumer of your classification, and get answers to the questions above. Or explain your probabilistic output to her or him, and let her or him walk through the next steps.

Improve this answer
$\endgroup$
  • 1
    $\begingroup$ Thank you very much for this insightful answer. I will further study the problem itself - I am certain I can somehow convert this property to the statistical learning part. $\endgroup$ – sdgaw erzswer Nov 6 '17 at 10:07
  • $\begingroup$ Wow, wish I could add something to this but got nothing, outstanding answer! $\endgroup$ – the_SJC Nov 6 '17 at 18:48
  • 6
    $\begingroup$ Very good answer: the questions are spot on! However, my profession being on the application side, whether finding a decision threshold is called statistics or not - it falls fully within my professional duties... And to me it is part of the model just like "pre-processing" is part of the model - also for the reason that all those decisions need to be covered in the validation process. $\endgroup$ – cbeleites unhappy with SX Nov 6 '17 at 21:49
11
$\begingroup$

Stephan's answer is great. It fundamentally depends on what you want to do with the classifier.

Just adding a few examples.

A way to find the best threshold is to define an objective function. For binary classification, this can be accuracy or F1-score for example. Depending on which you choose, the best threshold will be different. For F1-score, there is an interesting answer here: What is F1 Optimal Threshold? How to calculate it? . But saying "I want to use F1-score" is where you actually make the choice. Whether this choice is good or not depends on the final purpose.

Another way to see it is facing the trade-off between exploration and exploitation (Stephan's last point): The multi-armed bandit is an example of such a problem: you have to deal with two conflicting objectives of acquiring information and choosing the best bandit. One Bayesian strategy is to choose each bandit randomly with the probability it is the best. It's not exactly classification but dealing with output probabilities in a similar way.

If the classifier is just one brick in decision making algorithm, then the best threshold will depend on the final purpose of the algorithm. It should be evaluated and tuned in regard to the objective function of the whole process.

Improve this answer
$\endgroup$
  • $\begingroup$ Thank you for another great answer. If I understand correctly, if I am dealing with the final step in the pipeline, it is completely legitimate to directly optimize the threshold. $\endgroup$ – sdgaw erzswer Nov 6 '17 at 16:11
  • $\begingroup$ @sdgawerzswer: yes. And a) make sure you optimize the answer to the right question and b) make sure you validate that decision (and threshold finding) together with the rest of the model. $\endgroup$ – cbeleites unhappy with SX Nov 6 '17 at 21:51
3
$\begingroup$

There is possibly some value in considering how the probability is calculated. These days, Classifiers use a bias vector, which is multiplied by a matrix (linear algebra). As long as there are any non-zero values in the vector, the probability (the product of the vector and the matrix) will never be 0.

This causes confusion in the real world of people who didn't take linear algebra, I guess. They are bothered by the fact that there are probability scores for items that they think should have 0. In other words, they are confusing the statistical input, from the decision based on that input. As humans, we could say that something with a probability of 0.0002234 is the same as 0, in most "practical" use cases. In higher cognitive science discussions, maybe, there is an interesting discussion about why the bias vector does this, or rather, is this valid for cognitive applications.

Improve this answer
$\endgroup$
2
$\begingroup$

There is no wrong threshold. The threshold you choose depends of your objective in your prediction, or rather what you want to favor, for example precision versus recall (try to graph it and measure its associated AUC to compare different classification models of your choosing).

I am giving you this example of precision vs recall, because my own problem case i am working on right now, i choose my threshold depending of the minimal precision (or PPV Positive Predictive Value) i want my model to have when predicting, but i do not care much about negatives. As such i take the threshold that corresponds to the wanted precision once i have trained my model. Precision is my constraint and Recall is the performance of my model, when i compare to other classification models.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.