# Probability of random variable being another random variable

I am trying to understand the formalisation of the definition of calibration error of a classifier.

This definition is taken from Guo et al 2017 (https://arxiv.org/abs/1706.04599):

Let input $$X \in \mathcal{X}$$ and label $$Y \in \mathcal{Y}=\{1,2,...,K\}$$ be random variables. Let $$h$$ be a classifier with $$h(X)=(\hat{Y},\hat{P})$$ where r.v. $$\hat{Y}$$ is indicates the predicted class and r.v. $$\hat{P}$$ the model confidence. We define perfect calibration of $$h$$ as $$\mathbb{P}(\hat{Y}=Y|\hat{P}=p)=p, \forall p\in[0,1]$$ where the probability is taken over the joint distribution.

I do not understand the statement $$\mathbb{P}(\hat{Y}=Y|...)$$.

A) Do they mean $$\mathbb{P}(\hat{Y}=y|...)$$ or $$\mathbb{P}(\hat{Y}=y, Y=y|...)$$ (and do not distringuish between r.v. and realised values)? Or do they mean somthing like: $$F_Y(y)=F_\hat{Y}(y)$$ (i.e. $$\hat{Y} \stackrel{\mathcal{D}}{=} Y$$) with a probability $$p$$? Is it even well defined to have random variables that are only equal to each other with some probability < 1?

B) What do they mean with "probability is taken over the joint distribution"? Isn't this defining a conditional distribution? I only know this phrasing from the expectation which is an expression w.r.t. to a r.v. and you need to know of which variable you are integrating. Or are they merely trying to say: "this assumes the existence of a joint distribution $$\pi(\hat{Y}, \hat{P})$$"?

Edit: I am trying to understand why this definition is formally valid

A) The statement $$\mathbb{P}(\hat{Y}=Y|\hat{P}=p)=p, \forall p\in[0,1]$$, means predicted class's predicted probability is the actual probability. This includes misclassifications as well. No, it doesn't say anything about the joint of the predicted and realised $$(\hat{Y}, Y)$$ dependence. In practice, it only means the probabilities we get from softmax layer for example are indeed the probability of that class.
B) Yes, it is the joint distribution of $$(\hat{Y}, \hat{P})$$.
• thanks for your reply. My question was directed more towards the formal definition of this equation than what calibration means. How do you define a r.v. that is equal to another r.v. with some probability? what means taking a probability with respect to? does that mean we define a probability statement over a measure space that does not define $Y$? Oct 6, 2021 at 19:10