Reduce Classification Probability Threshold

Question

I have a question regarding classification in general. Let $f$ be a classifier, which outputs a set of probabilities given some data D. Normally, one would say: well, if $P(c|D) > 0.5$, we will assign a class 1, otherwise 0 (let this be a binary classification).

My question is, what if I find out, that if I classify the class as 1 also when the probabilities are larger than, for instance 0.2, and the classifier performs better. Is it legitimate to then use this new threshold when doing classification?

I would interpret the necessity for lower classification bound in the context of the data emitting a smaller signal; yet still significant for the classification problem.

I realize this is one way to do it. However, if this is not correct thinking of reducing the threshold, what would be some data transformations, which emphasize individual features in a similar manner, so that the threshold can remain at 0.5?

Answer 1

Frank Harrell has written about this on his blog: Classification vs. Prediction, which I agree with wholeheartedly.

Essentially, his argument is that the statistical component of your exercise ends when you output a probability for each class of your new sample. Choosing a threshold beyond which you classify a new observation as 1 vs. 0 is not part of the statistics any more. It is part of the decision component. And here, you need the probabilistic output of your model - but also considerations like:

• What are the consequences of deciding to treat a new observation as class 1 vs. 0? Do I then send out a cheap marketing mail to all 1s? Or do I apply an invasive cancer treatment with big side effects?
• What are the consequences of treating a "true" 0 as 1, and vice versa? Will I tick off a customer? Subject someone to unnecessary medical treatment?
• Are my "classes" truly discrete? Or is there actually a continuum (e.g., blood pressure), where clinical thresholds are in reality just cognitive shortcuts? If so, how far beyond a threshold is the case I'm "classifying" right now?
• Or does a low-but-positive probability to be class 1 actually mean "get more data", "run another test"?

So, to answer your question: talk to the end consumer of your classification, and get answers to the questions above. Or explain your probabilistic output to her or him, and let her or him walk through the next steps.

Answer 2

Stephan's answer is great. It fundamentally depends on what you want to do with the classifier.

Just adding a few examples.

A way to find the best threshold is to define an objective function. For binary classification, this can be accuracy or F1-score for example. Depending on which you choose, the best threshold will be different. For F1-score, there is an interesting answer here: What is F1 Optimal Threshold? How to calculate it? . But saying "I want to use F1-score" is where you actually make the choice. Whether this choice is good or not depends on the final purpose.

Another way to see it is facing the trade-off between exploration and exploitation (Stephan's last point): The multi-armed bandit is an example of such a problem: you have to deal with two conflicting objectives of acquiring information and choosing the best bandit. One Bayesian strategy is to choose each bandit randomly with the probability it is the best. It's not exactly classification but dealing with output probabilities in a similar way.

If the classifier is just one brick in decision making algorithm, then the best threshold will depend on the final purpose of the algorithm. It should be evaluated and tuned in regard to the objective function of the whole process.

Answer 3

There is possibly some value in considering how the probability is calculated. These days, Classifiers use a bias vector, which is multiplied by a matrix (linear algebra). As long as there are any non-zero values in the vector, the probability (the product of the vector and the matrix) will never be 0.

This causes confusion in the real world of people who didn't take linear algebra, I guess. They are bothered by the fact that there are probability scores for items that they think should have 0. In other words, they are confusing the statistical input, from the decision based on that input. As humans, we could say that something with a probability of 0.0002234 is the same as 0, in most "practical" use cases. In higher cognitive science discussions, maybe, there is an interesting discussion about why the bias vector does this, or rather, is this valid for cognitive applications.

Answer 4

There is no wrong threshold. The threshold you choose depends of your objective in your prediction, or rather what you want to favor, for example precision versus recall (try to graph it and measure its associated AUC to compare different classification models of your choosing).

I am giving you this example of precision vs recall, because my own problem case i am working on right now, i choose my threshold depending of the minimal precision (or PPV Positive Predictive Value) i want my model to have when predicting, but i do not care much about negatives. As such i take the threshold that corresponds to the wanted precision once i have trained my model. Precision is my constraint and Recall is the performance of my model, when i compare to other classification models.