Does probability calibration descrease model prediction variance? Does probability calibration decrease model prediction variance?
Example:
Let's say we have a classifier that is a mail spam detector. It outputs a score between 0-1 to quantify how likely a given email is to be a spam. Our local grocery store is innovative enough to have a newsletter. These emails are problematic since they routinely score between 0.8 and 0.99.
How would calibrating the classifier impact this interval? Could we reasonably expect that, after calibration, our model would be less variable about this newsletter predictions?
 A: It could be that these items are always likely to be spam and all should have spam probability scores above $0.9$ or even $0.99$. We would hope for the calibration to correct this, and such a result could squash the variance. From your description that there are many scores with modest values (say close to $0.5$), so it seems reasonable that your model might be underconfident, as the purple SVC example in the sklearn calibration link demonstrates. The calibration would allow your model to be more confident and make extreme predictions like $0.9$ and $0.99$. (Consider what calibration should do if every grocery store newsletter is spam, yet your spam scores are only 8/10 or 8.5/10. If you get a new grocery store newsletter, do you really think there's only an $85\%$ chance that it is spam?)
Alternatively, it could be that your model is overconfident. While some of those grocery store newsletters might be spam, maybe some really are not. Consequently, it could be that the calibration results in the interval being stretched to something like $0.3$-$0.9$, increasing the variability.
Consequently, without knowing more information, you cannot say either way what the calibration will do to your $0.8$-$0.99$ interval.
