I have one question about the evaluation metrics of classification models. I see many people report the precision and recall value for their classification models. Do they choose a threshold to convert predicted probability to predicted class and then calculate the precision and recall? If so, how do they choose the threshold?

If we compare the AUC value across different models that built for one dataset by different people, it's very direct and comparable. However, the precision and recall will vary based on the chosen threshold. Isn't this too arbitrary? If two people build classification models for one dataset, they both report their own precision and recall value, we'll not know who's model is better since they may use different threshold.

  • 1
    $\begingroup$ You can tune the threshold to maximize precision or recall, or a (weighted) combination between the two. Or you can use the default threshold, which is usually 0.5 (which usually makes no sense). If you see an analysis that does not explain how thresholds were chosen, that does not reflect well on the authors. If I see something like this in a science paper I am reviewing, I will absolutely require this is reported, and I may express some doubts about the entire analysis - if people misunderstand this, what else do they misunderstand? $\endgroup$ Commented May 16, 2023 at 19:43

1 Answer 1


You have identified a problem, yes. When you evaluate or compare models on their raw outputs (e.g., using log loss, Brier score, or AUC), you are comparing the models. When you evaluate models on precision, recall, or accuracy, you are comparing the models along with a decision rule (threshold), instead of comparing the models themselves.

This is among the reasons why statisticians see drawbacks to these threshold-based rules.

The most common decision rule is to classify according to the category given the highest probability. Even when the threshold is tuned, values like precision and recall make sense, so a model that does better on both of those does have some kind of advantage over a competitor that does worse on both, since you are evaluating the entire pipeline of model along with decision rule. Nonetheless, the raw outputs can be quite useful, and as is discussed in the link above, statisticians often advocate for the evaluation of those raw outputs in order to make optimal decisions. Stephan Kolassa’s answer to my question here gets into why and links to additional useful material.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.