-1
$\begingroup$

Setup

Let $f(\theta)$ be a prior on $\Theta$, and $X_1,\dots,X_n$ are iid according to $P_\theta$. From Bayes' rule, we derive the posterior as $$ g(\theta\mid x_1,\dots,x_n)=\frac{\Pr(x_1,\dots,x_n\mid\theta)f(\theta)}{\int_\Theta \Pr(x_1,\dots,x_n\mid\theta)f(\theta)d\theta} $$ where $\Pr(x_1,\dots,x_n\mid\theta)=P_\theta(x_1)\times\cdots\times P_\theta(x_n)$. Thus the $x$'s induce a distribution over the Bayesian posterior, with mean equal to the prior: $$ E_{\mathrm{x}_n}[g(\theta\mid\mathrm{x}_n)]=f(\theta) $$ where $\mathrm{x}_n=(x_1,\dots,x_n)$.

Questions

Is it true in general that the variance of the posterior is increasing in $n$?

Note that I'm NOT asking about whether the variance of $\theta$ computed using the posterior, i.e. $$Var(\theta)=\int(\theta-E(\theta))^2g(\theta\mid\mathrm{x}_n)d\theta,$$ would increase with $n$. This variance I know is decreasing in $n$. Rather, I'm asking whether the variance of the distribution over posteriors induced by $\mathrm{x}_n$ is increasing in $n$. In other word, is the following expression $$ E_{\mathrm{x}_n}[g(\theta\mid\mathrm{x}_n)-f(\theta)]^2 = E_{\mathrm{x}_n}[g^2(\theta\mid\mathrm{x}_n)]-f^2(\theta) $$ increasing in $n$? Since $n$ does not enter $f(\theta)$, my question amounts to whether $E_{\mathrm x_n}[g^2(\theta\mid \mathrm x_n)]$ is increasing in $n$?

What about the specific case where $f(\theta)$ and $P_\theta$ are both normal distributions (and thus $g(\theta\mid\mathrm{x}_n)$ is normal as well)?

$\endgroup$
1
$\begingroup$

Quite to the contrary, the variance of the posterior will decrease, as the posterior will concentrate more and more around the MLE as $n$ increases. This is known as "likelihood dominance", i.e. the data (likelihood) will "outvote" the effect of the prior in large samples.

$\endgroup$
  • $\begingroup$ I think you're talking about the variance of $\theta$ computed using the posterior, i.e. $Var(\theta)=\int (\theta-E(\theta))^2g(\theta\mid\mathrm{x}_n)d\theta$. What I'm asking about is something different. The signals $\mathrm{x}_n$ induce a distribution over the possible posteriors, and I'm asking whether the variance of this distribution (of posteriors) is increasing in $n$. $\endgroup$ – Herr K. Sep 14 '15 at 6:28
  • 2
    $\begingroup$ @HerrK.: I suggest you rephrase your question more precisely, as it is unclear as stated. $\endgroup$ – Xi'an Sep 14 '15 at 8:41
  • $\begingroup$ @Xi'an: question revised. $\endgroup$ – Herr K. Sep 14 '15 at 22:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.