# How to prove that, in finite sample, Bayesian posterior is more informative than the prior?

Suppose there is a space of possible models $$\theta \in \Theta$$, and that we can generate i.i.d. data $$\{x_1, x_2,...\}$$ from the true model.

Asymptotically, the Schwartz Theorem shows that Bayesian inference is consistent: the posterior will cluster around the true model.

However, can the same be shown in finite sample? That is, can we show that, in some appropriate sense, the posterior after one data observation $$P(\theta | x)$$ is (on average) more informative than the prior $$P(\theta)$$?

When the number of models is finite, a more concrete version of this question is: can we show that $$E(P(\theta^\text{true} | x)) > P(\theta^\text{true})$$?

If you want to pursue a result of this kind, you will need to be clear in how you measure the level of "informativeness" of the posterior. At the moment your measure of this is ambiguous --- you seem to be referring to the probability of getting the true parameter value, but in cases with a continuous prior, this will be zero under both the prior and posterior.

Now, assuming you are able to come up with a reasonable way to quantify this issue, you will still find that a prior may be better than a posterior in a finite sample, because of the possibility that the sampling distribution might be highly diffuse and the the prior could be really good. If you choose a prior that is sufficiently closely concentrated around the true parameter value, and if the sampling distribution is highly diffuse, then in small samples the posterior will usually move further away from the true value instead of closer to it. As the same size grows you will eventually get convergence due to the consisteny properties, but in small samples this might not occur.

In any case, if you want to construct a result like this, I recommend you start with some examples where you have a highly diffuse sampling distribution and a prior distribution that is closely concentrated around the true value of the parameter. Have a look at the properties of the posterior in this situation in order to get an idea of the limits or preliminary assumptions that would apply to the kind of result you want.

You need to define carefully what you mean by 'informative'. I'll give some examples with binomial experiments.

A common 'noninformative' prior for binomial success probability $$\theta$$ is the Jeffreys prior $$\mathsf{Beta}(.5,.5).$$ The density function is proportional to $$\theta^{.5-1}(1-\theta)^{.5-1}.$$

Then suppose you perform one Bernoulli trial, which results in success. The binomial likelihood is proportional to $$\theta^1(1-\theta)^0,$$ and by Bayes' Theorem the posterior distribution is $$\mathsf{Beta}(1.5, .5).$$

Alternatively, if you perform $$n=10$$ trials and get $$x= 7$$ successes, then the binomial likelihood is proportional to $$\theta^7(1-\theta)^3$$ and the posterior distribution is $$\mathsf{Beta}(7.5,3.5).$$

Plots of Jeffreys prior, the posterior after one trial, and the posterior after ten trials are shown in the figure below. Roughly speaking, one might say that beta distributions become increasingly informative as the sum of the two shape parameters increases.

R code for figure:

par(mfrow=c(1,3))
curve(dbeta(x,.5,.5), 0,1, lwd=2, ylim=c(0,3), col="blue",
ylab="Density", xlab="theta", main="BETA(.5,.5)")
abline(v=0:1, col="green"); abline(h=0, col="green")
curve(dbeta(x,1.5,.5), 0,1, lwd=2, ylim=c(0,3), col="blue",
ylab="Density", xlab="theta", main="BETA(1.5,.5)")
abline(v=0:1, col="green"); abline(h=0, col="green")
curve(dbeta(x,7.5,3.5), 0,1, lwd=2, ylim=c(0,3), col="blue",
ylab="Density", xlab="theta", main="BETA(7.5,3.5)")
abline(v=0:1, col="green"); abline(h=0, col="green")
par(mfrow=c(1,1))


For more detail on how a beta prior and a binomial likelihood yield a beta posterior, perhaps see this Q&A.

• Thanks BruceET. I think I'm simply looking for a proof that E(P(theta^true | x_1)) > P(theta^true).
– J Li
Commented Nov 1, 2021 at 6:36
• Doesn't seem quite right. In my Answ the prior and posterior means are $.5 < 7.5/11,$ respectively. But if there had been 3 successes and 7 failures out of 10 then the means would have been $.5 > 3.5/11.$ Commented Nov 1, 2021 at 6:45
• Yes, but my statement takes expectations over the possible realizations, no?
– J Li
Commented Nov 1, 2021 at 6:46
• Not sure I understand your notation. Commented Nov 1, 2021 at 6:48
• P(theta^true | x_1) is the posterior probability conditional on the realization of data x_1 (drawn from the data generating process theta^true). Then, take an expectation over all possible realizations x_1.
– J Li
Commented Nov 1, 2021 at 6:50

Okay, I think I have an answer now. Consider the case where both $$\theta$$ and $$x$$ can only take finite values. We want to show that $$E(P(\theta^\text{true} | x)) > P(\theta^\text{true})$$. Notice that:

\begin{align*} E(P(\theta^\text{true} | x)) & = E\left( \frac{P(x | \theta^\text{true}) P(\theta^\text{true})}{P(x)} \right) \\ & = P(\theta^\text{true}) \cdot \underbrace{E\left( \frac{P(x | \theta^\text{true})}{P(x)} \right)}_\text{Y} \end{align*}

Thus, we just need to show that $$Y > 1$$. Let's define $$\epsilon(x)$$ such that $$P(x | \theta_T) = P(x) + \epsilon(x)$$. Clearly, due to the property of probability mass functions, $$\sum_x \epsilon(x) = 0$$. Using this notation, it is easy to see that:

\begin{align*} Y & = \sum_x \frac{P^2(x | \theta^\text{true})}{P(x)} \\ & = \sum_x \frac{[P(x) + \epsilon(x)]^2}{P(x)} \\ & = \sum_x \left\{ P(x) + 2 \epsilon(x) + \frac{\epsilon^2(x)}{P(x)} \right\} \\ & = 1 + 0 + \sum_x \frac{\epsilon^2(x)}{P(x)} > 1 \end{align*}

where the last step assumes that $$\epsilon(x') \ne 0$$ for some $$x'$$. That is, the different models in $$\Theta$$ cannot have identical likelihoods.