My binary classifier machine learning pipelines are producing generalizable results in that they are predicting the positive class (probability > .5) with the same precision on unseen data in production as on my test set.
However, if I look more deeply into my production results, while this precision is consistent when probability > .5, the precision is extremely erratic when looking at higher classification thresholds above .5.
For example, probabilities above .6 gave me a precision of .77 on my test set, which was very encouraging, but give me precision less than .58 in production.
I am not sure what to make of these conflicting insights- what does it mean when a model can maintain its precision generally speaking, but not at specific probability thresholds? Should I be suspect of the entire model's viability? Is there anything I can do to ensure consistent results at different probability thresholds so that I can reliably limit false positives when setting the classification threshold higher?
Edit: To provide additional details:
- I have been training the models using the brier loss function. I tried the log loss function as well, and this improved things slightly, but the overall issue persists.
- The class membership is roughly balanced (53% belong to the positive class)
- The scenario in which I would adjust the probability threshold would look like this: I want the highest precision possible such that about 2% of my observations meet or surpass that probability threshold. So in the case that 53% of observations produce a probability > .5 that yields precision of 54%, 3% of observations produce a probability > .6 that yields a precision of 77%, and 1.5% of observations produce a probability > .65 yielding a precision of 85%, I would set the threshold to .6 such that the algorithm will only recognize 3% of observations as positive, even though it would only provide me with a precision of 77% on the test set vs. the 85% if I set the threshold to .65. In essence, I want the model that is exclusive enough to give me an acceptable precision while still identifying at least 3% of observations as being of the positive class.