# “forgetfulness” of the prior in the Bayesian setting?

It is well-known that as you have more evidence (say in the form of larger $n$ for $n$ i.i.d. examples), the Bayesian prior gets "forgotten", and most of the inference is impacted by the evidence (or the likelihood).

It is easy to see it for various specific case (such as Bernoulli with Beta prior or other type of examples) - but is there a way to see it in the general case with $x_1,\ldots,x_n \sim p(x|\mu)$ and some prior $p(\mu)$?

EDIT: I am guessing it cannot be shown in the general case for any prior (for example, a point-mass prior would keep the posterior a point-mass). But perhaps there are certain conditions under which a prior is forgotten.

Here is the kind of "path" I am thinking about showing something like that:

Assume the parameter space is $\Theta$, and let $p(\theta)$ and $q(\theta)$ be two priors which place non-zero probability mass on all of $\Theta$. So, the two posterior calculations for each prior amount to:

$$p(\theta | x_1,\ldots,x_n) = \frac{\prod_i p(x_i | \theta) p(\theta)}{\int_{\theta} \prod_i p(x_i | \theta) p(\theta) d\theta}$$

and $$q(\theta | x_1,\ldots,x_n) = \frac{\prod_i p(x_i | \theta) q(\theta)}{\int_{\theta} \prod_i p(x_i | \theta) q(\theta) d\theta}$$

If you divide $p$ by $q$ (the posteriors), then you get:

$$p(\theta | x_1,\ldots,x_n)/q(\theta | x_1,\ldots,x_n) = \frac{p(\theta)\int_{\theta} \prod_i p(x_i | \theta) q(\theta)d \theta}{q(\theta)\int_{\theta} \prod_i p(x_i | \theta) p(\theta)d \theta}$$

Now I would like to explore the above term as $n$ goes to $\infty$. Ideally it would go to $1$ for a certain $\theta$ that "makes sense" or some other nice behavior, but I can't figure out how to show anything there.

-
For some intuition, note that the likelihood scales with the sample size while the prior does not. – Macro Jan 6 '13 at 21:47
@Macro, thanks, I also had that intuition, but I couldn't push it further. See my edits above. – bayesianOrFrequentist Jan 6 '13 at 22:20
The first few chapters of Ghosh and Ramamoorthi's textbook Bayesian Nonparametrics fleshes out the sorts of things you are talking about (at first in a parametric setting, then a nonparametric one); it is available through Springer online for free if you are at an appropriate institution. There are multiple ways of formalizing the lack of dependence on the prior asymptotically, but of course there are a few regularity conditions. – guy Jan 6 '13 at 23:12
Note that the posterior ratio is just proportional to the prior ratio, so the likelihood nor evidence ratio doesn't really influence this. – probabilityislogic Mar 8 '13 at 22:14

Just a rough, but hopefully intuitive answer.

1. Look at it from the log-space point of view: $$-\log P(\theta|x_1, \ldots, x_n) = -\log P(\theta) -\sum_{i=1}^n \log P(x_i|\theta) - C_n$$ where $C_n>0$ is a constant that depends on the data, but not on the parameter, and where your likelihoods assume i.i.d. observations. Hence, just concentrate on the part that determines the shape of your posterior, namely $$S_n = -\log P(\theta) -\sum_{i=1}^n \log P(x_i|\theta)$$

2. Assume that there is a $D>0$ such that $-\log P(\theta) \leq D$. This is reasonable for discrete distributions.

3. Since the terms are all positive, $S_n$ "will" grow (I'm skipping the technicalities here). But the contribution of the prior is bounded by $D$. Hence, the fraction contributed by the prior, which is at most $D/S_n$, decreases monotonically with each additional observation.

Rigorous proofs of course have to face the technicalities (and they can be very difficult), but the setting above is IMHO the very basic part.

-

I am somewhat confused by what the statements the "prior is forgetten" and "most of the inference is impacted by the evidence" are supposed to mean. I assume you mean as the amount of data increases, the (sequence of) estimator(s) approaches the true value of the parameter regardless of our prior.

Assuming some regularity conditions on the form of the posterior distribution, Bayes Estimators are consistent and asymptotically unbiased (see Gelman et al, chapter 4). This means as the sample size increases the bayes estimator approaches the true value of the parameter. Consistency means the bayes estimator converges in probability to the true parameter value and asymptotic unbiasedness means that, assuming $\theta_0$ is the true value of the parameter,

$$\frac{E[\hat{\theta}|\theta_0]-\theta_0}{\sqrt{\mathrm{Var}(\hat{\theta})}}\overset{p}\rightarrow0$$

The convergence does not depend on the specific form of the prior, but only that the posterior distribution obtained from the prior and the likelihood satisfy the regularity conditions.

The most important regularity condition mentioned in Gelman et al is that the likelihood be a continuous function of the parameter and the true value of the parameter be in the interior of the parameter space. Also, as you noted, the posterior must be nonzero in an open neighborhood of the true value of the true value of the parameter. Usually, your prior should be nonzero on the entire parameter space.

-
thanks, very insightful. I was hoping actually for a result that would not even relate to the "true" parameter value. Just showing that technically, as you have more evidence, the posterior you are going to get is the same irregardless of the prior that you started with. I am going to make some edits to reflect that. – bayesianOrFrequentist Jan 6 '13 at 22:13
@bayesianOrFrequentist Take a look at the so-called Bayesian central limit theorem. – Stéphane Laurent Mar 8 '13 at 9:36