Precision and recall reported in classification model

Question

I have one question about the evaluation metrics of classification models. I see many people report the precision and recall value for their classification models. Do they choose a threshold to convert predicted probability to predicted class and then calculate the precision and recall? If so, how do they choose the threshold?

If we compare the AUC value across different models that built for one dataset by different people, it's very direct and comparable. However, the precision and recall will vary based on the chosen threshold. Isn't this too arbitrary? If two people build classification models for one dataset, they both report their own precision and recall value, we'll not know who's model is better since they may use different threshold.

Answer 1

You have identified a problem, yes. When you evaluate or compare models on their raw outputs (e.g., using log loss, Brier score, or AUC), you are comparing the models. When you evaluate models on precision, recall, or accuracy, you are comparing the models along with a decision rule (threshold), instead of comparing the models themselves.

This is among the reasons why statisticians see drawbacks to these threshold-based rules.

The most common decision rule is to classify according to the category given the highest probability. Even when the threshold is tuned, values like precision and recall make sense, so a model that does better on both of those does have some kind of advantage over a competitor that does worse on both, since you are evaluating the entire pipeline of model along with decision rule. Nonetheless, the raw outputs can be quite useful, and as is discussed in the link above, statisticians often advocate for the evaluation of those raw outputs in order to make optimal decisions. Stephan Kolassa’s answer to my question here gets into why and links to additional useful material.