1
$\begingroup$

Does probability calibration decrease model prediction variance?

Example: Let's say we have a classifier that is a mail spam detector. It outputs a score between 0-1 to quantify how likely a given email is to be a spam. Our local grocery store is innovative enough to have a newsletter. These emails are problematic since they routinely score between 0.8 and 0.99.

How would calibrating the classifier impact this interval? Could we reasonably expect that, after calibration, our model would be less variable about this newsletter predictions?

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ Less indecisive and less variable need not go together. Always predicting $0.5$ is totally indecisive but not at all variable, while your scores above $0.8$ are relatively decisive but also variable. // Are the newsletters typically spam? // When you do calibration, why do you do it? Are your predictions too confident? Not confident enough? All over the place? The purple SVM and orange naïve Bayes plots in the sklearn link indicate quite different issues. $\endgroup$
    – Dave
    Commented Feb 1, 2023 at 15:23
  • $\begingroup$ @Dave, agreed about the indecisive/variable part, will edit that out soon. // The example is made up to illustrate the concept // Typically calibration is performed to have model output a probability rather than a score. As I said, the example is made up, but the model that I have doesn't have too confident predictions. There is a reasonable number of both high and low predictions. $\endgroup$
    – Glue
    Commented Feb 1, 2023 at 15:38

1 Answer 1

Reset to default
2
$\begingroup$

It could be that these items are always likely to be spam and all should have spam probability scores above $0.9$ or even $0.99$. We would hope for the calibration to correct this, and such a result could squash the variance. From your description that there are many scores with modest values (say close to $0.5$), so it seems reasonable that your model might be underconfident, as the purple SVC example in the sklearn calibration link demonstrates. The calibration would allow your model to be more confident and make extreme predictions like $0.9$ and $0.99$. (Consider what calibration should do if every grocery store newsletter is spam, yet your spam scores are only 8/10 or 8.5/10. If you get a new grocery store newsletter, do you really think there's only an $85\%$ chance that it is spam?)

Alternatively, it could be that your model is overconfident. While some of those grocery store newsletters might be spam, maybe some really are not. Consequently, it could be that the calibration results in the interval being stretched to something like $0.3$-$0.9$, increasing the variability.

Consequently, without knowing more information, you cannot say either way what the calibration will do to your $0.8$-$0.99$ interval.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.