I am trying to understand the formalisation of the definition of calibration error of a classifier.

This definition is taken from Guo et al 2017 (https://arxiv.org/abs/1706.04599):

Let input $X \in \mathcal{X}$ and label $Y \in \mathcal{Y}=\{1,2,...,K\}$ be random variables. Let $h$ be a classifier with $h(X)=(\hat{Y},\hat{P})$ where r.v. $\hat{Y}$ is indicates the predicted class and r.v. $\hat{P}$ the model confidence. We define perfect calibration of $h$ as $$\mathbb{P}(\hat{Y}=Y|\hat{P}=p)=p, \forall p\in[0,1]$$ where the probability is taken over the joint distribution.

I do not understand the statement $\mathbb{P}(\hat{Y}=Y|...)$.

A) Do they mean $\mathbb{P}(\hat{Y}=y|...)$ or $\mathbb{P}(\hat{Y}=y, Y=y|...)$ (and do not distringuish between r.v. and realised values)? Or do they mean somthing like: $F_Y(y)=F_\hat{Y}(y)$ (i.e. $\hat{Y} \stackrel{\mathcal{D}}{=} Y$) with a probability $p$? Is it even well defined to have random variables that are only equal to each other with some probability < 1?

B) What do they mean with "probability is taken over the joint distribution"? Isn't this defining a conditional distribution? I only know this phrasing from the expectation which is an expression w.r.t. to a r.v. and you need to know of which variable you are integrating. Or are they merely trying to say: "this assumes the existence of a joint distribution $\pi(\hat{Y}, \hat{P})$"?

Edit: I am trying to understand why this definition is formally valid


1 Answer 1


This is still an intuitive definition.

A) The statement $\mathbb{P}(\hat{Y}=Y|\hat{P}=p)=p, \forall p\in[0,1]$, means predicted class's predicted probability is the actual probability. This includes misclassifications as well. No, it doesn't say anything about the joint of the predicted and realised $(\hat{Y}, Y)$ dependence. In practice, it only means the probabilities we get from softmax layer for example are indeed the probability of that class.

Let's say if 0.6 is outputted for that class (imagine set of instances having 0.6 for that class). On the unseen data if we predict that class, it should be correct 60% of the time.

B) Yes, it is the joint distribution of $(\hat{Y}, \hat{P})$.

  • $\begingroup$ thanks for your reply. My question was directed more towards the formal definition of this equation than what calibration means. How do you define a r.v. that is equal to another r.v. with some probability? what means taking a probability with respect to? does that mean we define a probability statement over a measure space that does not define $Y$? $\endgroup$ Oct 6, 2021 at 19:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.