0
$\begingroup$

I'm reading through a text that is explaining calibration curves, and the following description is provided:

To be well-calibrated, the probabilities must effectively reflect the true likelihood of the event of interest. Returning to the spam filter illustration, if a model produces a probability or probability-like value of 20 % for the likelihood of a particular e-mail to be spam, then this value would be well-calibrated if similar types of messages would truly be from that class on average in 1 of 5 samples.

What I don't understand is the "similar types of messages" part. How do calibration curves determine similarity between observations?

$\endgroup$

1 Answer 1

1
$\begingroup$

Similarity is determined by the classifier that is being calibrated. This paragraph is saying that if we run our classifier on a pile of emails, then for all of the emails $x$ that we predict $p(y=\text{SPAM}\mid x)=0.2$ (i.e., emails that our classifier scores similarly), roughly 20% of them should actually be $\text{SPAM}$.

It may be helpful to think about calibrated outputs as an interpretation of a classification decision. It doesn't really add information to the decision function, it simply maps it to the interval $[0,1]$ so it can be interpreted as a probability.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.