"forgetfulness" of the prior in the Bayesian setting?

Question

It is well-known that as you have more evidence (say in the form of larger $n$ for $n$ i.i.d. examples), the Bayesian prior gets "forgotten", and most of the inference is impacted by the evidence (or the likelihood).

It is easy to see it for various specific case (such as Bernoulli with Beta prior or other type of examples) - but is there a way to see it in the general case with $x_1,\ldots,x_n \sim p(x|\mu)$ and some prior $p(\mu)$?

EDIT: I am guessing it cannot be shown in the general case for any prior (for example, a point-mass prior would keep the posterior a point-mass). But perhaps there are certain conditions under which a prior is forgotten.

Here is the kind of "path" I am thinking about showing something like that:

Assume the parameter space is $\Theta$, and let $p(\theta)$ and $q(\theta)$ be two priors which place non-zero probability mass on all of $\Theta$. So, the two posterior calculations for each prior amount to:

$$p(\theta | x_1,\ldots,x_n) = \frac{\prod_i p(x_i | \theta) p(\theta)}{\int_{\theta} \prod_i p(x_i | \theta) p(\theta) d\theta}$$

and $$q(\theta | x_1,\ldots,x_n) = \frac{\prod_i p(x_i | \theta) q(\theta)}{\int_{\theta} \prod_i p(x_i | \theta) q(\theta) d\theta}$$

If you divide $p$ by $q$ (the posteriors), then you get:

$$p(\theta | x_1,\ldots,x_n)/q(\theta | x_1,\ldots,x_n) = \frac{p(\theta)\int_{\theta} \prod_i p(x_i | \theta) q(\theta)d \theta}{q(\theta)\int_{\theta} \prod_i p(x_i | \theta) p(\theta)d \theta}$$

Now I would like to explore the above term as $n$ goes to $\infty$. Ideally it would go to $1$ for a certain $\theta$ that "makes sense" or some other nice behavior, but I can't figure out how to show anything there.

Answer 1

Just a rough, but hopefully intuitive answer.

1. Look at it from the log-space point of view: $$ -\log P(\theta|x_1, \ldots, x_n) = -\log P(\theta) -\sum_{i=1}^n \log P(x_i|\theta) - C_n $$ where $C_n>0$ is a constant that depends on the data, but not on the parameter, and where your likelihoods assume i.i.d. observations. Hence, just concentrate on the part that determines the shape of your posterior, namely $$ S_n = -\log P(\theta) -\sum_{i=1}^n \log P(x_i|\theta) $$

2. Assume that there is a $D>0$ such that $-\log P(\theta) \leq D$. This is reasonable for discrete distributions.

3. Since the terms are all positive, $S_n$ "will" grow (I'm skipping the technicalities here). But the contribution of the prior is bounded by $D$. Hence, the fraction contributed by the prior, which is at most $D/S_n$, decreases monotonically with each additional observation.

Rigorous proofs of course have to face the technicalities (and they can be very difficult), but the setting above is IMHO the very basic part.

Answer 2

I am somewhat confused by what the statements the "prior is forgetten" and "most of the inference is impacted by the evidence" are supposed to mean. I assume you mean as the amount of data increases, the (sequence of) estimator(s) approaches the true value of the parameter regardless of our prior.

Assuming some regularity conditions on the form of the posterior distribution, Bayes Estimators are consistent and asymptotically unbiased (see Gelman et al, chapter 4). This means as the sample size increases the bayes estimator approaches the true value of the parameter. Consistency means the bayes estimator converges in probability to the true parameter value and asymptotic unbiasedness means that, assuming $\theta_0$ is the true value of the parameter,

$$ \frac{E[\hat{\theta}|\theta_0]-\theta_0}{\sqrt{\mathrm{Var}(\hat{\theta})}}\overset{p}\rightarrow0 $$

The convergence does not depend on the specific form of the prior, but only that the posterior distribution obtained from the prior and the likelihood satisfy the regularity conditions.

The most important regularity condition mentioned in Gelman et al is that the likelihood be a continuous function of the parameter and the true value of the parameter be in the interior of the parameter space. Also, as you noted, the posterior must be nonzero in an open neighborhood of the true value of the true value of the parameter. Usually, your prior should be nonzero on the entire parameter space.