# Does probability calibration descrease model prediction variance?

Does probability calibration decrease model prediction variance?

Example: Let's say we have a classifier that is a mail spam detector. It outputs a score between 0-1 to quantify how likely a given email is to be a spam. Our local grocery store is innovative enough to have a newsletter. These emails are problematic since they routinely score between 0.8 and 0.99.

How would calibrating the classifier impact this interval? Could we reasonably expect that, after calibration, our model would be less variable about this newsletter predictions?

• Less indecisive and less variable need not go together. Always predicting $0.5$ is totally indecisive but not at all variable, while your scores above $0.8$ are relatively decisive but also variable. // Are the newsletters typically spam? // When you do calibration, why do you do it? Are your predictions too confident? Not confident enough? All over the place? The purple SVM and orange naïve Bayes plots in the sklearn link indicate quite different issues.
– Dave
Commented Feb 1, 2023 at 15:23
• @Dave, agreed about the indecisive/variable part, will edit that out soon. // The example is made up to illustrate the concept // Typically calibration is performed to have model output a probability rather than a score. As I said, the example is made up, but the model that I have doesn't have too confident predictions. There is a reasonable number of both high and low predictions.
– Glue
Commented Feb 1, 2023 at 15:38

It could be that these items are always likely to be spam and all should have spam probability scores above $$0.9$$ or even $$0.99$$. We would hope for the calibration to correct this, and such a result could squash the variance. From your description that there are many scores with modest values (say close to $$0.5$$), so it seems reasonable that your model might be underconfident, as the purple SVC example in the sklearn calibration link demonstrates. The calibration would allow your model to be more confident and make extreme predictions like $$0.9$$ and $$0.99$$. (Consider what calibration should do if every grocery store newsletter is spam, yet your spam scores are only 8/10 or 8.5/10. If you get a new grocery store newsletter, do you really think there's only an $$85\%$$ chance that it is spam?)
Alternatively, it could be that your model is overconfident. While some of those grocery store newsletters might be spam, maybe some really are not. Consequently, it could be that the calibration results in the interval being stretched to something like $$0.3$$-$$0.9$$, increasing the variability.
Consequently, without knowing more information, you cannot say either way what the calibration will do to your $$0.8$$-$$0.99$$ interval.