Setup

Let $f(\theta)$ be a prior on $\Theta$, and $X_1,\dots,X_n$ are iid according to $P_\theta$. From Bayes' rule, we derive the posterior as $$g(\theta\mid x_1,\dots,x_n)=\frac{\Pr(x_1,\dots,x_n\mid\theta)f(\theta)}{\int_\Theta \Pr(x_1,\dots,x_n\mid\theta)f(\theta)d\theta}$$ where $\Pr(x_1,\dots,x_n\mid\theta)=P_\theta(x_1)\times\cdots\times P_\theta(x_n)$. Thus the $x$'s induce a distribution over the Bayesian posterior, with mean equal to the prior: $$E_{\mathrm{x}_n}[g(\theta\mid\mathrm{x}_n)]=f(\theta)$$ where $\mathrm{x}_n=(x_1,\dots,x_n)$.

Questions

Is it true in general that the variance of the posterior is increasing in $n$?

Note that I'm NOT asking about whether the variance of $\theta$ computed using the posterior, i.e. $$Var(\theta)=\int(\theta-E(\theta))^2g(\theta\mid\mathrm{x}_n)d\theta,$$ would increase with $n$. This variance I know is decreasing in $n$. Rather, I'm asking whether the variance of the distribution over posteriors induced by $\mathrm{x}_n$ is increasing in $n$. In other word, is the following expression $$E_{\mathrm{x}_n}[g(\theta\mid\mathrm{x}_n)-f(\theta)]^2 = E_{\mathrm{x}_n}[g^2(\theta\mid\mathrm{x}_n)]-f^2(\theta)$$ increasing in $n$? Since $n$ does not enter $f(\theta)$, my question amounts to whether $E_{\mathrm x_n}[g^2(\theta\mid \mathrm x_n)]$ is increasing in $n$?

What about the specific case where $f(\theta)$ and $P_\theta$ are both normal distributions (and thus $g(\theta\mid\mathrm{x}_n)$ is normal as well)?

Quite to the contrary, the variance of the posterior will decrease, as the posterior will concentrate more and more around the MLE as $n$ increases. This is known as "likelihood dominance", i.e. the data (likelihood) will "outvote" the effect of the prior in large samples.

• I think you're talking about the variance of $\theta$ computed using the posterior, i.e. $Var(\theta)=\int (\theta-E(\theta))^2g(\theta\mid\mathrm{x}_n)d\theta$. What I'm asking about is something different. The signals $\mathrm{x}_n$ induce a distribution over the possible posteriors, and I'm asking whether the variance of this distribution (of posteriors) is increasing in $n$. – Herr K. Sep 14 '15 at 6:28
• @HerrK.: I suggest you rephrase your question more precisely, as it is unclear as stated. – Xi'an Sep 14 '15 at 8:41
• @Xi'an: question revised. – Herr K. Sep 14 '15 at 22:16