9
$\begingroup$

What are the necessary conditions for a model's posterior to converge to a point mass in the limit of infinite observations? What is an example that breaks this convergence result?

Off the top of my head, I think misspecified models or nonidentifiable models would break these convergence guarantees, but how do I go about formalizing this?

Edit: for those who voted to close this because the question is ambiguous, please comment below for how I can resolve your concern.

$\endgroup$
  • $\begingroup$ Be more specific which posterior's convergence $\endgroup$ – develarist Oct 7 at 4:08
  • $\begingroup$ hi @develarist thanks for pointing out that my question is ambiguous, but what do you mean by "which posterior's convergence"? I am asking in general, when the posterior distribution of a statistical model converges in distribution to a point mass. Maybe my questions were misleading due to the inconsistent of pluralization? I've revised the question with that in mind. $\endgroup$ – user228809 Oct 7 at 6:30
5
$\begingroup$

Convergence of the posterior due to convergence of the likelihood

One way to look at 'convergence' is in a frequentist way, for increasing sample size the posterior will, with increasing probability, be high for the true parameter and low for the false parameter.

For this we can use the Bayes factor

$$\frac{P(\theta_1\vert x)}{P(\theta_0\vert x)} = \frac{P(x \vert \theta_1)}{P(x \vert \theta_0)} \frac{P(\theta_1)}{P(\theta_0)} $$

where $\theta_0$ is the true parameter value and $\theta_1$ is any other alternative value. (maybe it is a bit strange to speak about the true parameter in a Bayesian context, but maybe the same is true for speaking about converging of the posterior, which is maybe more like a frequentist property of the posterior)

Assume that the likelihood ratio ${P(x \vert \theta_1)}/{P(x \vert \theta_0)}$ will converge to 0 in probability for all values $\theta_1$ that do not have a likelihood function that is the same as the likelihood function for the true parameter value $\theta_0$. (we will show that later)

So if ${P(x \vert \theta_1)}/{P(x \vert \theta_0)}$ converges, and if $P(\theta_0)$ is nonzero, then you will have that ${P(\theta_1\vert x)}/{P(\theta_0\vert x)}$ converges. And this implies that $P(x \vert \theta)$ converges to / concentrates in the point $\theta_0$.

What are the necessary conditions for a model's posterior to converge to a point mass in the limit of infinite observations?

So you need two conditions:

  • The likelihood function of two different parameters must be different.

  • $P(\theta)$ is non-zero for the correct $\theta$. (you can argue similarly for densities $f(\theta)$ as prior)

    Intuitive: If your prior gives zero density/probability to the true $\theta$ then the posterior will never give a non-zero density/probability to the true $\theta$, no matter how large sample you take.


Convergence of the likelihood ratio to zero

The likelihood ratio of a sample of size $n$ converges to zero (when $\theta_1$ is not the true parameter).

$$ \frac{P(x_1, x_2, \dots , x_n \vert \theta_1)}{P(x_1, x_2, \dots , x_n \vert \theta_0)} \quad \xrightarrow{P} \quad 0$$

or for the negative log-likelihood ratio

$$-\Lambda_{\theta_1,n} = - \log \left( \frac{P(x_1, x_2, \dots , x_n \vert \theta_1)}{P(x_1, x_2, \dots , x_n \vert \theta_0)} \right) \quad \xrightarrow{P} \quad \infty$$

We can show this by using the law of large numbers (and we need to assume that the measurements are independent).

If we assume that the measurements are independent then we can view the log-likelihood for a sample of size $n$ as the sum of the values of the log-likelihood for single measurements

$$\Lambda_{\theta_1,n} = \log \left( \frac{P(x_1, x_2, \dots , x_n \vert \theta_1)}{P(x_1, x_2, \dots , x_n \vert \theta_0)} \right) = \log \left( \prod_{i=1}^n \frac{P(x_i \vert \theta_1)}{P(x_i \vert \theta_0)} \right) = \sum_{i=1}^n \log \left( \frac{P(x_i \vert \theta_1)}{P(x_i \vert \theta_0)} \right)$$

Note that the expectation value of the negative log-likelihood

$$E\left[- \log \left( \frac{P_{x \vert \theta_1}(x \vert \theta_1)}{P_{x \vert \theta_0}(x \vert \theta_0)} \right)\right] = -\sum_{ x \in \chi} P_{x \vert \theta_0}(x \vert \theta_0) \log \left( \frac{P_{x \vert \theta_1}(x \vert \theta_1)}{P_{x \vert \theta_0}(x \vert \theta_0)} \right) \geq 0$$

resembles the Kullback-Leibler divergence, which is positive as can be shown by Gibbs' inequality, and equality to zero occurs iff $P(x \vert \theta_1) = P(x \vert \theta_0)$:

So if this expectation is positive then by the law of large numbers, $-{\Lambda_{\theta_1,n}}/{n}$ convergences to some positive constant $c$

$$\lim_{n \to \infty} P\left( \left| -\frac{\Lambda_{\theta_1,n}}{n}-c \right| > \epsilon \right) = 0$$

which implies that $-{\Lambda_{\theta_1,n}}$ will converge to infinity. For any $K>0$

$$\lim_{n \to \infty} P\left( {-\Lambda_{\theta_1,n}} < K \right) = 0$$

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Arguing via CLT is not really necessary (and may require unnecessarily strong moment assumptions, as you noted). For convenience, define $\Lambda$ instead via the reciprocal of the likelihood ratio. Then, by LLN, $\frac{\Lambda}{n}$ converges in probability to the KL divergence, which is strictly positive. So $\Lambda \rightarrow \infty$ in probability, which would prove the convergence claim when the parameter space has two elements. $\endgroup$ – Michael Oct 7 at 19:39
  • $\begingroup$ Thanks for your answer. I like this approach and justification; i'll leave this unanswered for a few more days. $\endgroup$ – user228809 Oct 7 at 20:10
  • $\begingroup$ @Michael, I was thinking of using the law of large numbers. But it is $\Lambda/n$ that converges and not $\Lambda$. $\endgroup$ – Sextus Empiricus Oct 7 at 20:27
  • $\begingroup$ "But it is Λ/n that converges and not Λ..."---yes, that is all that is needed (for this argument, in the case of two-element parameter space). The general argument uses the fact that the posterior densities form a martingale, which imposes no rate whatsoever (Doob's martingale convergence theorem). This suggests that an argument that insists on a $\sqrt{n}$-rate---such as one that relies on the CLT---is not likely to work in general. (Asymptotic normality also extraneous here.) $\endgroup$ – Michael Oct 8 at 18:29
  • $\begingroup$ @Michael you are right. The convergence of the likelihood can be shown by LLN. For some reason I had mixed up my mind. All we need is that the probability of the log-likelihood being above some number can be made as small we want. I thought that the scaling from $\Lambda/n$ to $\Lambda$ was mixing this up, but it doesn't matter at all. $\endgroup$ – Sextus Empiricus Oct 8 at 19:44
5
$\begingroup$

Adding three points to the answer by @SextusEmpiricus:

First, Doob's Theorem says that the posterior (under correct model specification) converges to the truth except on a set of parameters $\theta$ with prior probability zero. In a finite-dimensional setting you would typically have a prior that puts some mass everywhere, so that a set with prior probability zero also has Lebesgue measure zero.

Second, finite-dimensional misspecified models will typically also have (frequentist) posterior convergence to a point mass, at the $\theta_0$ which minimises the Kullback-Leibler divergence to the data-generating model. The arguments for this are analogous to the arguments for convergence of misspecified MLEs to the 'least false' model, and can be done along the lines of @SextusEmpiricus's answer.

Third, this is all much more complicated for infinite-dimensional parameters, partly because sets of prior probability 1 can be quite small in infinite-dimensional spaces. For any specified $\epsilon>0$, a probability distribution places at least $1-\epsilon$ of its mass on some compact set $K_\epsilon$. In, eg, Hilbert or Banach spaces a compact set can't contain any open ball.

In infinite-dimensional problems:

  • Doob's Theorem is still true, but it's less useful.
  • Whether or not the posterior converges to a point depends on how big (flexible, overfitting,..) the model is
  • It's quite possible for a correctly specified model to have a prior converging to the wrong point mass. In fact, Freedman gave a reasonable-looking problem for which this is typical. So prior choice is more tricky than it is in finite-dimensional problems.
| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ "In, eg, Hilbert or Banach spaces a compact set can't contain any open ball..."---that depends on the topology. Closed balls are weak-* compact in Banach spaces. In Freedman's example, the parameter space is the positive part of unit ball of $l^1$, but the topology considered is the weak-* topology (equivalently, topology of weak convergence), not the Banach space topology. In particular, the parameter space itself is compact. It seems that the weak-* topology is the natural one for this context in general. $\endgroup$ – Michael Oct 8 at 1:33
  • $\begingroup$ Yes, you can define a different topology to make any set you like compact. There's still a valid problem that big spaces are big, completely separate from the Freedman problem. $\endgroup$ – Thomas Lumley Oct 8 at 2:05
  • $\begingroup$ What would be an example of a Bayesian problem, where the parameter space---subset of a Banach space (say)---is considered with the norm topology? If the normal means model, where the parameter lies in the Hilbert space $l^2$, has been considered from the Bayesian perspective (?), I guess that would be one example. $\endgroup$ – Michael Oct 8 at 2:34
  • $\begingroup$ When I look up Doob's Theorem and follow the link to Wikipedia, then I get to en.wikipedia.org/wiki/Doob%27s_martingale_convergence_theorems which is not the typical easy reading wiki article. Do you know of another good introductory reference? $\endgroup$ – Sextus Empiricus Oct 8 at 9:50
  • $\begingroup$ If the prior does not contain the true parameter or is a false model, than the convergence can be to a multi modal distribution. A set of several points that minimize the divergence. stats.stackexchange.com/questions/468227/… (I know, you wrote typical, here is just an example how it can fail to go to a point mass) $\endgroup$ – Sextus Empiricus Oct 8 at 21:52
2
$\begingroup$

The necessary and sufficient condition that the posterior converges to the point mass at the true parameter is that the model is correctly specified and identified, for any prior whose support contains the true parameter.

(Convergence here means that, under the law determined by $\theta$, for every neighborhood $U$ of $\theta$, the measure $\mu_n(U)$ of $U$ under posterior $\mu_n$ converges almost surely to $1$.)

Below is a simple argument for the case of finite parameter spaces, say $\{\theta_0, \theta_1\}$.

(The argument can be extended to the general case. The general statement is that consistency holds except on a set of prior measure zero. The assumption that the parameter space is finite avoids measure-theoretic considerations. The general statement comes with the usual caveat for almost-everywhere statements---one cannot say whether it holds for a given $\theta$.)

Necessity

Suppose the posterior is consistent at $\theta_0$. Then it's immediate that the model must be identified. Otherwise, the likelihood ratio process $$ \prod_{k = 1}^n \frac{p(x_k|\theta_1)}{p(x_k|\theta_0)}, \, n = 1, 2, \cdots $$ equals $1$ almost surely and the posterior is equal to the prior for all $n$, almost surely.

Sufficiency

Now suppose the posterior is consistent. This implies that the likelihood ratio process converges to zero almost surely.

Two things to notice here:

  1. Under the law determined by $\theta_0$, the likelihood ratio process $$ M_n = \prod_{k = 1}^n \frac{p(x_k|\theta_1)}{p(x_k|\theta_0)} \equiv \prod_{k = 1}^n X_k. $$ is a nonnegative martingale, and, by the consistency assumption, $M_n \stackrel{a.s.}{\rightarrow} M_{\infty} \equiv 0$.

  2. $p(x|\theta_1)$ is equal to $p(x|\theta_0)$ $dx$-almost everywhere with respect to reference measure $dx$ if and only if $\rho = \int \sqrt{ p(x|\theta_1) p(x|\theta_0)} dx = 1$. In general, $0 \leq \rho \leq 1$.

Define $$ N_n = \prod_{k = 1}^n \frac{ X_k^{\frac12} }{\rho}= \frac{1}{\rho^n} \prod_{k = 1}^n X_k^{\frac12}, $$ which is also a nonnegative martingale.

Now suppose model is not identified, i.e. $\rho = 1$. Then $(N_n)$ is uniformly bounded in $L^1$ (because $E[N_n^2] = 1$ for all $n$). By Doob's $L^2$ inequality, $$ E[\, \sup_n M_n\, ] \leq 4 \sup_n E[\, N_n^2 \,] < \infty. $$ This implies that $(X_n)$ is a uniformly integrable martingale. By Doob's convergence theorem for UI martingale, $M_n = E[M_{\infty}|M_k, k \leq n] = 0$, which is impossible---$\prod_{k=1}^n p(x_k|\theta_1)$ cannot be zero almost surely if $\rho = 1$.

Comments on Sufficiency

Couple comments on the sufficiency part:

  1. The coefficient $\rho$ was first considered by Kakutani (1948), who used it to prove the consistency of the LR test, among other things.

  2. For finite parameter space, sufficiency can also be shown via the KL-divergence argument in the answer of @SextusEmpiricus (although I don't believe that argument extends to the general setting; the martingale property seems more primitive). In the case of finite parameter space, both arguments make use of convexity (via the $\log$ and $\sqrt{\cdot}$ functions respectively.)

Infinite Dimensional Parameter Space

The set of priors whose support contains the true parameter can be "very small", when the parameter space is infinite dimensional. In the example of Freedman (1965), mentioned by @ThomasLumley, the parameter space $\Theta$ is the set of all probability measures on $\mathbb{N}$, i.e. $$ \Theta = \{ (p_i)_{i \geq 1}: \; p_i \geq 0 \; \forall i, \mbox{ and } \sum_i p_i = 1\} \subset l^1(\mathbb{N}), $$ and given the weak-* topology induced by the pairing between $l^{\infty}$ and $l^1$. The set of priors is the set of probability measures on $\Theta$, given the topology of weak convergence. Freedman showed that the (true parameter, prior)-pairs which are consistent is "small" with respect to the product topology.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.