My machine learning models have reproducible results generally, but not at specific classification thresholds. What does this mean?

Question

My binary classifier machine learning pipelines are producing generalizable results in that they are predicting the positive class (probability > .5) with the same precision on unseen data in production as on my test set.

However, if I look more deeply into my production results, while this precision is consistent when probability > .5, the precision is extremely erratic when looking at higher classification thresholds above .5.

For example, probabilities above .6 gave me a precision of .77 on my test set, which was very encouraging, but give me precision less than .58 in production.

I am not sure what to make of these conflicting insights- what does it mean when a model can maintain its precision generally speaking, but not at specific probability thresholds? Should I be suspect of the entire model's viability? Is there anything I can do to ensure consistent results at different probability thresholds so that I can reliably limit false positives when setting the classification threshold higher?

Edit: To provide additional details:

• I have been training the models using the brier loss function. I tried the log loss function as well, and this improved things slightly, but the overall issue persists.
• The class membership is roughly balanced (53% belong to the positive class)
• The scenario in which I would adjust the probability threshold would look like this: I want the highest precision possible such that about 2% of my observations meet or surpass that probability threshold. So in the case that 53% of observations produce a probability > .5 that yields precision of 54%, 3% of observations produce a probability > .6 that yields a precision of 77%, and 1.5% of observations produce a probability > .65 yielding a precision of 85%, I would set the threshold to .6 such that the algorithm will only recognize 3% of observations as positive, even though it would only provide me with a precision of 77% on the test set vs. the 85% if I set the threshold to .65. In essence, I want the model that is exclusive enough to give me an acceptable precision while still identifying at least 3% of observations as being of the positive class.

Answer 1

This report ("Loss Functions for Binary Class Probability Estimation and Classification: Structure and Applications" by Andreas Buja, Werner Stuetzle, Yi Shen) discusses in detail how different loss functions interact with different choices of probability cutoffs. As the report notes, when you change the probability cutoff you are effectively (whether consciously or not) changing your estimate of the relative costs of false-negative and false-positive classifications.

The report goes into detail about how different proper scoring rules can provide different performance depending on the region of interest along the probability scale. For example, the log-loss puts more dramatic emphasis on the extremes of the probability scale than does the Brier loss. So perhaps it isn't surprising that a log-loss-based model performed better than the Brier-based model as you moved toward more extreme probability cutoffs away from 0.5. The report describes a family of proper scoring rules that could be used to tweak your model toward the probability range of ultimate interest.

I discuss this a bit further in another answer.