7
$\begingroup$

In the book Bayesian Data Analysis, the authors state on page 128:

The concept of calibration of the posterior mean [is] the Bayesian analogue to the classical notion of bias.

They define the miscalibration of the posterior mean of some parameter $\theta$ as $$ \text{miscalibration} = E(\theta|\hat{\theta}) - \hat{\theta} $$ where $$ \hat{\theta} = E(\theta|y) $$ and $y$ is the observed data.

Can someone clarify what this means? How does this $E(\theta|\hat{\theta})$ term work? It's a conditional expectation that's conditional on something that is itself a conditional expectation. How does that work? Can you show that the miscalibration is 0 if the prior distribution is true, i.e. if the data are constructed by first drawing $\theta$ from $p(\theta)$ and then drawing $y$ from $p(y|\theta)$?

$\endgroup$

1 Answer 1

6
$\begingroup$

It may be confusing that the $\hat \theta = E[\theta \mid y]$ seems to use the chosen prior and the posterior distribution then implied by the observation, while the $E\left[\theta \mid \hat \theta\right]-\hat \theta$ in the miscalibration calculation seems to use the actual (though presumably unknown) distribution for $\theta$. It is probably easier with an example.

Suppose we have $\theta \in [0,1]$ as the parameter of a Bernoulli random variable $Y$ which is $1$ with probability $\theta$ and $0$ otherwise

If we use a prior $p(\theta)=2\theta$, then

  • observing $Y=1$ will suggest to us a posterior distribution $p(\theta \mid Y=1)=3\theta^2$ with mean $\hat \theta =\frac34$,
  • observing $Y=0$ will suggest to us a posterior distribution $p(\theta \mid Y=0)=6\theta(1-\theta)$ with mean $\hat \theta =\frac12$,

and there is a $1-1$ relationship here between the value of $Y$ and $\hat \theta$.

If the prior is the correct distribution of $\theta$ then

  • when $\hat \theta =\frac34$, we have $E\left[\theta \mid \hat \theta=\frac34\right] - \hat\theta =\frac34-\frac34=0$

  • when $\hat \theta =\frac12$, we have $E\left[\theta \mid \hat \theta=\frac12\right] - \hat\theta =\frac12-\frac12=0$

automatically as asserted, and also

  • the probability of observing $Y=1$ and estimating $\hat \theta=\frac34$ is $\frac23$
  • the probability of observing $Y=0$ and estimating $\hat \theta=\frac12$ is $\frac13$

so $E\left[\hat\theta\right]=\frac23 \times \frac34 +\frac13\times \frac12 =\frac23 = E[\theta]$ and all is well with the world.

But if the prior were wrong and in fact $\theta$ in reality had a density of $2(1-\theta)$ i.e. with $\theta$ and $Y$ more likely to be smaller than the assumed prior suggested, then the actual conditional expectations for $\theta$ would be smaller

  • when $\hat \theta =\frac34$, we would have $E\left[\theta \mid \hat \theta=\frac34\right] - \hat\theta =\frac12-\frac34=-\frac14$
  • when $\hat \theta =\frac12$, we would have $E\left[\theta \mid \hat \theta=\frac12\right] - \hat\theta =\frac14-\frac12=-\frac14$

so a miscalibration of $- \frac14$ (in more complicated cases it does not have to be constant) and

  • the probability of observing $Y=1$ and estimating $\hat \theta=\frac34$ would have been $\frac13$
  • the probability of observing $Y=0$ and estimating $\hat \theta=\frac12$ would have been $\frac23$

making $E[\hat\theta]=\frac13 \times \frac34 +\frac23\times \frac12 =\frac7{12} \not = \frac13 = E[\theta]$, which may be undesirable.

$\endgroup$
4
  • $\begingroup$ I still don't understand how you calculate $E(\theta | \hat{\theta})$. For example above you simply state that in your first example $E(\theta | \hat{\theta} = \frac{3}{4}) = \frac{3}{4}$. But I just fundamentally don't understand how such an expectation conditional on another expectation works. Could you perhaps elaborate this in baby steps? I don't see where that conditional comes in, or what probability distribution is used exactly to calculate the expectation. $\endgroup$
    – Willem
    Dec 20, 2021 at 13:54
  • $\begingroup$ @Willem If your prior is $p(\theta)=2\theta$ then, having observed $Y=1$, your posterior would be $p(\theta \mid Y=1) = 3\theta^2$, leading to $\hat \theta = \mathbb E[\theta \mid Y=1]= \frac34$. But if instead your observe $Y=0$ you get $\hat \theta = \frac12$. So with this prior, the event $Y=1$ is equivalent to the event $\hat \theta = \frac34$ while the event $Y=0$ is equivalent to the event $\hat \theta = \frac12$. If the prior is correct then you get $E[\theta \mid \hat \theta =\frac34] =\frac34$ $\endgroup$
    – Henry
    Dec 29, 2021 at 22:58
  • $\begingroup$ Thank you for your reply. Am I correct in understanding that this means conditioning on $\hat{\theta}$ is equivalent to conditioning on $Y$? The way I'm understanding it now is that really miscalibration = $E_{\text{true prior}}(\theta | y) - E_{\text{chosen prior}}(\theta | y)$. Is this interpretation correct? $\endgroup$
    – Willem
    Jan 10, 2022 at 20:29
  • $\begingroup$ @Willem - yes - I think that is a way of looking at it $\endgroup$
    – Henry
    Jan 10, 2022 at 20:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.