# Classification probability threshold

I have a question regarding classification in general. Let f be a classifier, which outputs a set of probabilities given some data D. Normally, one would say: well, if P(c|D) > 0.5, we will assign a class 1, otherwise 0 (let this be a binary classification).

My question is, what if I find out, that if I classify as 1 also the probabilities, larger than: i.e. 0.2, the classifier performs better. Is it legitimate to then use this new threshold when doing classification?

I would interpret the necessity for lower classification bound in the context of the data emitting a smaller signal; yet still significant for the classification problem.

I realize this is one way to do it, yet if this is not correct thinking, what would be some data transformations, which emphasize individual features in a similar manner, so that the threshold can remain at 0.5?

• You already have some excellent answer, so let me just say this: your "normally" is not a normally that should be normal. I'm not sure where the "threshold at 0.5" thing became standard, and I know there is some, otherwise excellent, software that encourages the idea, but it's a very poor practice in general. – Matthew Drury Nov 6 '17 at 15:25
• @MatthewDrury: unless, of course, the score is the well-calibrated relevant posterior probability of making no important mistake (the latter would take care of different costs of misclassification). – cbeleites Nov 6 '17 at 21:43

Frank Harrell has written about this on his blog: Classification vs. Prediction, which I agree with wholeheartedly.

Essentially, his argument is that the statistical component of your exercise ends when you output a probability for each class of your new sample. Choosing a threshold beyond which you classify a new observation as 1 vs. 0 is not part of the statistics any more. It is part of the decision component. And here, you need the probabilistic output of your model - but also considerations like:

• What are the consequences of deciding to treat a new observation as class 1 vs. 0? Do I then send out a cheap marketing mail to all 1s? Or do I apply an invasive cancer treatment with big side effects?
• What are the consequences of treating a "true" 0 as 1, and vice versa? Will I tick off a customer? Subject someone to unnecessary medical treatment?
• Are my "classes" truly discrete? Or is there actually a continuum (e.g., blood pressure), where clinical thresholds are in reality just cognitive shortcuts? If so, how far beyond a threshold is the case I'm "classifying" right now?
• Or does a low-but-positive probability to be class 1 actually mean "get more data", "run another test"?

So, to answer your question: talk to the end consumer of your classification, and get answers to the questions above. Or explain your probabilistic output to her or him, and let her or him walk through the next steps.

• Thank you very much for this insightful answer. I will further study the problem itself - I am certain I can somehow convert this property to the statistical learning part. – sdgaw erzswer Nov 6 '17 at 10:07
• Wow, wish I could add something to this but got nothing, outstanding answer! – the_SJC Nov 6 '17 at 18:48
• Very good answer: the questions are spot on! However, my profession being on the application side, whether finding a decision threshold is called statistics or not - it falls fully within my professional duties... And to me it is part of the model just like "pre-processing" is part of the model - also for the reason that all those decisions need to be covered in the validation process. – cbeleites Nov 6 '17 at 21:49

There is possibly some value in considering how the probability is calculated. These days, Classifiers use a bias vector, which is multiplied by a matrix (linear algebra). As long as there are any non-zero values in the vector, the probability (the product of the vector and the matrix) will never be 0.

This causes confusion in the real world of people who didn't take linear algebra, I guess. They are bothered by the fact that there are probability scores for items that they think should have 0. In other words, they are confusing the statistical input, from the decision based on that input. As humans, we could say that something with a probability of 0.0002234 is the same as 0, in most "practical" use cases. In higher cognitive science discussions, maybe, there is an interesting discussion about why the bias vector does this, or rather, is this valid for cognitive applications.

There is no wrong threshold. The threshold you choose depends of your objective in your prediction, or rather what you want to favor, for example precision versus recall (try to graph it and measure its associated AUC to compare different classification models of your choosing).

I am giving you this example of precision vs recall, because my own problem case i am working on right now, i choose my threshold depending of the minimal precision (or PPV Positive Predictive Value) i want my model to have when predicting, but i do not care much about negatives. As such i take the threshold that corresponds to the wanted precision once i have trained my model. Precision is my constraint and Recall is the performance of my model, when i compare to other classification models.

Stephan's answer is great. It fundamentally depends on what you want to do with the classifier.