# Please clarify Bayesian calibration of the posterior mean

In the book Bayesian Data Analysis, the authors state on page 128:

The concept of calibration of the posterior mean [is] the Bayesian analogue to the classical notion of bias.

They define the miscalibration of the posterior mean of some parameter $$\theta$$ as $$\text{miscalibration} = E(\theta|\hat{\theta}) - \hat{\theta}$$ where $$\hat{\theta} = E(\theta|y)$$ and $$y$$ is the observed data.

Can someone clarify what this means? How does this $$E(\theta|\hat{\theta})$$ term work? It's a conditional expectation that's conditional on something that is itself a conditional expectation. How does that work? Can you show that the miscalibration is 0 if the prior distribution is true, i.e. if the data are constructed by first drawing $$\theta$$ from $$p(\theta)$$ and then drawing $$y$$ from $$p(y|\theta)$$?

It may be confusing that the $$\hat \theta = E[\theta \mid y]$$ seems to use the chosen prior and the posterior distribution then implied by the observation, while the $$E\left[\theta \mid \hat \theta\right]-\hat \theta$$ in the miscalibration calculation seems to use the actual (though presumably unknown) distribution for $$\theta$$. It is probably easier with an example.

Suppose we have $$\theta \in [0,1]$$ as the parameter of a Bernoulli random variable $$Y$$ which is $$1$$ with probability $$\theta$$ and $$0$$ otherwise

If we use a prior $$p(\theta)=2\theta$$, then

• observing $$Y=1$$ will suggest to us a posterior distribution $$p(\theta \mid Y=1)=3\theta^2$$ with mean $$\hat \theta =\frac34$$,
• observing $$Y=0$$ will suggest to us a posterior distribution $$p(\theta \mid Y=0)=6\theta(1-\theta)$$ with mean $$\hat \theta =\frac12$$,

and there is a $$1-1$$ relationship here between the value of $$Y$$ and $$\hat \theta$$.

If the prior is the correct distribution of $$\theta$$ then

• when $$\hat \theta =\frac34$$, we have $$E\left[\theta \mid \hat \theta=\frac34\right] - \hat\theta =\frac34-\frac34=0$$

• when $$\hat \theta =\frac12$$, we have $$E\left[\theta \mid \hat \theta=\frac12\right] - \hat\theta =\frac12-\frac12=0$$

automatically as asserted, and also

• the probability of observing $$Y=1$$ and estimating $$\hat \theta=\frac34$$ is $$\frac23$$
• the probability of observing $$Y=0$$ and estimating $$\hat \theta=\frac12$$ is $$\frac13$$

so $$E\left[\hat\theta\right]=\frac23 \times \frac34 +\frac13\times \frac12 =\frac23 = E[\theta]$$ and all is well with the world.

But if the prior were wrong and in fact $$\theta$$ in reality had a density of $$2(1-\theta)$$ i.e. with $$\theta$$ and $$Y$$ more likely to be smaller than the assumed prior suggested, then the actual conditional expectations for $$\theta$$ would be smaller

• when $$\hat \theta =\frac34$$, we would have $$E\left[\theta \mid \hat \theta=\frac34\right] - \hat\theta =\frac12-\frac34=-\frac14$$
• when $$\hat \theta =\frac12$$, we would have $$E\left[\theta \mid \hat \theta=\frac12\right] - \hat\theta =\frac14-\frac12=-\frac14$$

so a miscalibration of $$- \frac14$$ (in more complicated cases it does not have to be constant) and

• the probability of observing $$Y=1$$ and estimating $$\hat \theta=\frac34$$ would have been $$\frac13$$
• the probability of observing $$Y=0$$ and estimating $$\hat \theta=\frac12$$ would have been $$\frac23$$

making $$E[\hat\theta]=\frac13 \times \frac34 +\frac23\times \frac12 =\frac7{12} \not = \frac13 = E[\theta]$$, which may be undesirable.

• I still don't understand how you calculate $E(\theta | \hat{\theta})$. For example above you simply state that in your first example $E(\theta | \hat{\theta} = \frac{3}{4}) = \frac{3}{4}$. But I just fundamentally don't understand how such an expectation conditional on another expectation works. Could you perhaps elaborate this in baby steps? I don't see where that conditional comes in, or what probability distribution is used exactly to calculate the expectation. Dec 20, 2021 at 13:54
• @Willem If your prior is $p(\theta)=2\theta$ then, having observed $Y=1$, your posterior would be $p(\theta \mid Y=1) = 3\theta^2$, leading to $\hat \theta = \mathbb E[\theta \mid Y=1]= \frac34$. But if instead your observe $Y=0$ you get $\hat \theta = \frac12$. So with this prior, the event $Y=1$ is equivalent to the event $\hat \theta = \frac34$ while the event $Y=0$ is equivalent to the event $\hat \theta = \frac12$. If the prior is correct then you get $E[\theta \mid \hat \theta =\frac34] =\frac34$ Dec 29, 2021 at 22:58
• Thank you for your reply. Am I correct in understanding that this means conditioning on $\hat{\theta}$ is equivalent to conditioning on $Y$? The way I'm understanding it now is that really miscalibration = $E_{\text{true prior}}(\theta | y) - E_{\text{chosen prior}}(\theta | y)$. Is this interpretation correct? Jan 10, 2022 at 20:29
• @Willem - yes - I think that is a way of looking at it Jan 10, 2022 at 20:54