# Posterior variance vs variance of the posterior mean

This question is about the frequentist properties of Bayesian methods.

Suppose we have data $${\bf y}$$ generated from a distribution with a single parameter $$\theta$$, equipped with a prior $$\pi(\theta)$$. This leads to a posterior distribution $$\pi(\theta|{\bf y})$$. A Bayesian might characterize the uncertainty surrounding $$\theta$$ using the posterior variance

\begin{align} \mathbb{V}(\theta|{\bf y}) = \int_\Theta \left(\theta-\hat\theta\right)^2\pi(\theta|{\bf y}) \ d\theta &&&&&& (1) \end{align}

where $$\hat\theta$$ is the posterior mean. A frequentist wearing his Bayesian hat (or perhaps the other way around) might choose to focus on the posterior mean (i.e. an estimate of $$\theta$$) more directly. The variance of the posterior mean is given by

\begin{align} \mathbb{V}(\hat\theta | \theta) = \int_\Theta\left(\hat\theta - E(\hat\theta)\right)^2 f(\hat\theta|\theta) \ d\hat\theta &&&&&& (2) \end{align}

If the posterior mean is unbiased a consistent estimator for $$\theta$$, and the posterior distribution $$\pi(\theta|{\bf y})$$ "looks like" the sampling distribution $$f(\hat\theta|\theta)$$, then equations (1) and (2) start to look alike (asymptotically).

Why might we care: Thinking about the frequentist coverage of credible intervals (mixing Bayesian and frequentist worlds again), it seems that the ratio of these two quantities should provide valuable information. For example, if the posterior variance is much smaller than the variance of the posterior mean, coverage of the credible intervals may be poor (less than nominal).

1. Does it even make sense to compare these quantities? What kind of information can be gained from comparing them?
2. Can anybody point me to some references dealing with this specific issue? (I've tried searching this, but can't find anything on this site or otherwise dealing with this specific issue).
• A Bayes estimator like a posterior mean is always biased. Commented Sep 20, 2019 at 4:46
• 1. When they're not equivalent, it could be a case of model misspecification. 2. arxiv.org/abs/1806.00550 Commented Oct 15, 2020 at 17:43
• @Xi'an Good point thank you. I will update the phrasing to use the term consistent. Asymptotically, my question still holds. Commented Jan 2 at 20:15

There is no particular reason that $$\pi(\theta|\mathbf{y})$$ should look anything like $$f(\hat{\theta}|\theta)$$ as functions of $$\theta$$. The latter is a sampling distribution for the parameter estimator $$\hat{\theta}$$, which may have a substantially different form to the likelihood function for the data $$\mathbf{y}$$. Moreover, the parameter estimator in Bayesian analysis is hardly ever unbiased, since it incorporates a prior distribution. For those reasons I think it is extremely unlikely that the integrands in these variance equations are going to "look like" each other, except perhaps in unusual special cases.

1. Does it even make sense to compare these quantities? What kind of information can be gained from comparing them?

Since these are variances for entirely different quantities, conditional on different things, any useful comparison is probably going to come through some corresponding interval estimator for $$\theta$$. The posterior standard deviation $$\mathbb{S}(\theta|\mathbf{y})$$ ought to give you a rough idea of the width of a credible interval for $$\theta$$, whereas the standard error $$\mathbb{S}(\hat{\theta}|\theta)$$ ought to give you a rough idea of the width of a confidence interval for $$\theta$$.

Consequently, if you were willing to discard "shape information" for the distributions then you could reasonably say that the comparison of the two variances will give you an idea of the relative accuracy of the credible interval versus the confidence interval. This would be a bit tentative in my view, but it might be possible under some simplifying assumptions.

2 . Can anybody point me to some references dealing with this specific issue? (I've tried searching this, but can't find anything on this site or otherwise dealing with this specific issue).

I'm not familiar with any literature on this topic, but perhaps you could search for comparisons of accuracy/width of credible intervals and confidence intervals. If there is any literature on that subject then I imagine it will involve these two variance quantities.

• Thank you for this answer. You say "...you could reasonably say that the comparison of the two variances will give. you an idea of the relative accuracy of the credible interval versus the confidence interval". This is the heart of my question!! This seems like it should be a useful comparison. In what settings does this provide useful information? Commented Jan 2 at 20:19
• There is a whole subfield of Bayesian statistics that looks at the frequentist properties of Bayesian estimators and compares them with classical estimators. I imagine that this kind of inquiry might be useful there.
– Ben
Commented Jan 2 at 20:57