When do posteriors converge to a point mass?

Question

What are the necessary conditions for a model's posterior to converge to a point mass in the limit of infinite observations? What is an example that breaks this convergence result?

Off the top of my head, I think misspecified models or nonidentifiable models would break these convergence guarantees, but how do I go about formalizing this?

Edit: for those who voted to close this because the question is ambiguous, please comment below for how I can resolve your concern.

Answer 1

Convergence of the posterior due to convergence of the likelihood

One way to look at 'convergence' is in a frequentist way, for increasing sample size the posterior will, with increasing probability, be high for the true parameter and low for the false parameter.

For this we can use the Bayes factor

$$\frac{P(\theta_1\vert x)}{P(\theta_0\vert x)} = \frac{P(x \vert \theta_1)}{P(x \vert \theta_0)} \frac{P(\theta_1)}{P(\theta_0)} $$

where $\theta_0$ is the true parameter value and $\theta_1$ is any other alternative value. (maybe it is a bit strange to speak about the true parameter in a Bayesian context, but maybe the same is true for speaking about converging of the posterior, which is maybe more like a frequentist property of the posterior)

Assume that the likelihood ratio ${P(x \vert \theta_1)}/{P(x \vert \theta_0)}$ will converge to 0 in probability for all values $\theta_1$ that do not have a likelihood function that is the same as the likelihood function for the true parameter value $\theta_0$. (we will show that later)

So if ${P(x \vert \theta_1)}/{P(x \vert \theta_0)}$ converges, and if $P(\theta_0)$ is nonzero, then you will have that ${P(\theta_1\vert x)}/{P(\theta_0\vert x)}$ converges. And this implies that $P(x \vert \theta)$ converges to / concentrates in the point $\theta_0$.

What are the necessary conditions for a model's posterior to converge to a point mass in the limit of infinite observations?

So you need two conditions:

• The likelihood function of two different parameters must be different.

• $P(\theta)$ is non-zero for the correct $\theta$. (you can argue similarly for densities $f(\theta)$ as prior)

  Intuitive: If your prior gives zero density/probability to the true $\theta$ then the posterior will never give a non-zero density/probability to the true $\theta$, no matter how large sample you take.

---

Convergence of the likelihood ratio to zero

The likelihood ratio of a sample of size $n$ converges to zero (when $\theta_1$ is not the true parameter).

$$ \frac{P(x_1, x_2, \dots , x_n \vert \theta_1)}{P(x_1, x_2, \dots , x_n \vert \theta_0)} \quad \xrightarrow{P} \quad 0$$

or for the negative log-likelihood ratio

$$-\Lambda_{\theta_1,n} = - \log \left( \frac{P(x_1, x_2, \dots , x_n \vert \theta_1)}{P(x_1, x_2, \dots , x_n \vert \theta_0)} \right) \quad \xrightarrow{P} \quad \infty$$

We can show this by using the law of large numbers (and we need to assume that the measurements are independent).

If we assume that the measurements are independent then we can view the log-likelihood for a sample of size $n$ as the sum of the values of the log-likelihood for single measurements

$$\Lambda_{\theta_1,n} = \log \left( \frac{P(x_1, x_2, \dots , x_n \vert \theta_1)}{P(x_1, x_2, \dots , x_n \vert \theta_0)} \right) = \log \left( \prod_{i=1}^n \frac{P(x_i \vert \theta_1)}{P(x_i \vert \theta_0)} \right) = \sum_{i=1}^n \log \left( \frac{P(x_i \vert \theta_1)}{P(x_i \vert \theta_0)} \right)$$

Note that the expectation value of the negative log-likelihood

$$E\left[- \log \left( \frac{P_{x \vert \theta_1}(x \vert \theta_1)}{P_{x \vert \theta_0}(x \vert \theta_0)} \right)\right] = -\sum_{ x \in \chi} P_{x \vert \theta_0}(x \vert \theta_0) \log \left( \frac{P_{x \vert \theta_1}(x \vert \theta_1)}{P_{x \vert \theta_0}(x \vert \theta_0)} \right) \geq 0$$

resembles the Kullback-Leibler divergence, which is positive as can be shown by Gibbs' inequality, and equality to zero occurs iff $P(x \vert \theta_1) = P(x \vert \theta_0)$:

So if this expectation is positive then by the law of large numbers, $-{\Lambda_{\theta_1,n}}/{n}$ convergences to some positive constant $c$

$$\lim_{n \to \infty} P\left( \left| -\frac{\Lambda_{\theta_1,n}}{n}-c \right| > \epsilon \right) = 0$$

which implies that $-{\Lambda_{\theta_1,n}}$ will converge to infinity. For any $K>0$

$$\lim_{n \to \infty} P\left( {-\Lambda_{\theta_1,n}} < K \right) = 0$$

Answer 2

Adding three points to the answer by @SextusEmpiricus:

First, Doob's Theorem says that the posterior (under correct model specification) converges to the truth except on a set of parameters $\theta$ with prior probability zero. In a finite-dimensional setting you would typically have a prior that puts some mass everywhere, so that a set with prior probability zero also has Lebesgue measure zero.

Second, finite-dimensional misspecified models will typically also have (frequentist) posterior convergence to a point mass, at the $\theta_0$ which minimises the Kullback-Leibler divergence to the data-generating model. The arguments for this are analogous to the arguments for convergence of misspecified MLEs to the 'least false' model, and can be done along the lines of @SextusEmpiricus's answer.

Third, this is all much more complicated for infinite-dimensional parameters, partly because sets of prior probability 1 can be quite small in infinite-dimensional spaces. For any specified $\epsilon>0$, a probability distribution places at least $1-\epsilon$ of its mass on some compact set $K_\epsilon$. In, eg, Hilbert or Banach spaces a compact set can't contain any open ball.

In infinite-dimensional problems:

• Doob's Theorem is still true, but it's less useful.
• Whether or not the posterior converges to a point depends on how big (flexible, overfitting,..) the model is
• It's quite possible for a correctly specified model to have a prior converging to the wrong point mass. In fact, Freedman gave a reasonable-looking problem for which this is typical. So prior choice is more tricky than it is in finite-dimensional problems.

Answer 3

The necessary and sufficient condition that the posterior converges to the point mass at the true parameter is that the model is correctly specified and identified, for any prior whose support contains the true parameter.

(Convergence here means that, under the law determined by $\theta$, for every neighborhood $U$ of $\theta$, the measure $\mu_n(U)$ of $U$ under posterior $\mu_n$ converges almost surely to $1$.)

Below is a simple argument for the case of finite parameter spaces, say $\{\theta_0, \theta_1\}$.

(The argument can be extended to the general case. The general statement is that consistency holds except on a set of prior measure zero. The assumption that the parameter space is finite avoids measure-theoretic considerations. The general statement comes with the usual caveat for almost-everywhere statements---one cannot say whether it holds for a given $\theta$.)

Necessity

Suppose the posterior is consistent at $\theta_0$. Then it's immediate that the model must be identified. Otherwise, the likelihood ratio process $$ \prod_{k = 1}^n \frac{p(x_k|\theta_1)}{p(x_k|\theta_0)}, \, n = 1, 2, \cdots $$ equals $1$ almost surely and the posterior is equal to the prior for all $n$, almost surely.

Sufficiency

Now suppose the posterior is consistent. This implies that the likelihood ratio process converges to zero almost surely.

Two things to notice here:

1. Under the law determined by $\theta_0$, the likelihood ratio process $$ M_n = \prod_{k = 1}^n \frac{p(x_k|\theta_1)}{p(x_k|\theta_0)} \equiv \prod_{k = 1}^n X_k. $$ is a nonnegative martingale, and, by the consistency assumption, $M_n \stackrel{a.s.}{\rightarrow} M_{\infty} \equiv 0$.

2. $p(x|\theta_1)$ is equal to $p(x|\theta_0)$ $dx$-almost everywhere with respect to reference measure $dx$ if and only if $\rho = \int \sqrt{ p(x|\theta_1) p(x|\theta_0)} dx = 1$. In general, $0 \leq \rho \leq 1$.

Define $$ N_n = \prod_{k = 1}^n \frac{ X_k^{\frac12} }{\rho}= \frac{1}{\rho^n} \prod_{k = 1}^n X_k^{\frac12}, $$ which is also a nonnegative martingale.

Now suppose model is not identified, i.e. $\rho = 1$. Then $(N_n)$ is uniformly bounded in $L^1$ (because $E[N_n^2] = 1$ for all $n$). By Doob's $L^2$ inequality, $$ E[\, \sup_n M_n\, ] \leq 4 \sup_n E[\, N_n^2 \,] < \infty. $$ This implies that $(X_n)$ is a uniformly integrable martingale. By Doob's convergence theorem for UI martingale, $M_n = E[M_{\infty}|M_k, k \leq n] = 0$, which is impossible---$\prod_{k=1}^n p(x_k|\theta_1)$ cannot be zero almost surely if $\rho = 1$.

Comments on Sufficiency

Couple comments on the sufficiency part:

1. The coefficient $\rho$ was first considered by Kakutani (1948), who used it to prove the consistency of the LR test, among other things.

2. For finite parameter space, sufficiency can also be shown via the KL-divergence argument in the answer of @SextusEmpiricus (although I don't believe that argument extends to the general setting; the martingale property seems more primitive). In the case of finite parameter space, both arguments make use of convexity (via the $\log$ and $\sqrt{\cdot}$ functions respectively.)

Infinite Dimensional Parameter Space

The set of priors whose support contains the true parameter can be "very small", when the parameter space is infinite dimensional. In the example of Freedman (1965), mentioned by @ThomasLumley, the parameter space $\Theta$ is the set of all probability measures on $\mathbb{N}$, i.e. $$ \Theta = \{ (p_i)_{i \geq 1}: \; p_i \geq 0 \; \forall i, \mbox{ and } \sum_i p_i = 1\} \subset l^1(\mathbb{N}), $$ and given the weak-* topology induced by the pairing between $l^{\infty}$ and $l^1$. The set of priors is the set of probability measures on $\Theta$, given the topology of weak convergence. Freedman showed that the (true parameter, prior)-pairs which are consistent is "small" with respect to the product topology.