8
$\begingroup$

Consider the posterior density function given (as usual) by $$ \pi(\theta) \prod_{i=1}^n f(x_i;\theta),$$ with prior density $\pi$ and distribution $f(\cdot;\theta)$ of the $n$ observations $x_1, \dots, x_n$, conditional on the parameter value $\theta$.

Under certain conditions, the posterior distribution is asymptotically normal (a result which is known as the Bernstein-von Mises theorem, see e.g. v.d Vaart, Asymptotic Statistics, Section 10.2, for rigorous arguments, or Young & Smith, Essentials of Statistical Inference, Section 9.12, for an informal discussion.)

Are there any (hopefully elementary) examples in which the Bayesian posterior is not asymptotically normal? In particular are there examples where

  1. $\pi$ and $f$ are continuously differentiable with respect to $\theta$?
  2. $\pi(\theta) > 0$ for all $\theta$?

One example I noted in the literature is that where $X_1, \dots, X_n$ are independent Cauchy random variables with location parameter $\theta$. In this case, with positive probability there exist multiple local maxima of the likelihood function (See Young & Smith, Example 8.3). Perhaps this may present a problem in the B-vM theorem although I am not sure.

Update: Sufficient conditions for BvM are (as stated in v.d Vaart, Section 10.2):

  • data is obtained from distribution with fixed parameter $\theta_0$

  • experiment is `differentiable in quadratic mean' at $\theta_0$ with non-singular Fisher information matrix $I(\theta_0)$

  • the prior is absolutely continuous in a region around $\theta_0$

  • the model is continuous and identifiable

  • there exists a test which separates $H_0 : \theta = \theta_0$ from $H_1 : \|\theta -\theta_0\| \geq \varepsilon$ for some $\varepsilon > 0$

$\endgroup$
  • $\begingroup$ I think it is more relevant to whether the KL support of prior contains the TRUE parameter? $\endgroup$ – Henry.L Mar 4 '17 at 3:51
2
+50
$\begingroup$

1.Does the Cauchy example contradicts the Bernstein von-Mises Theorem?

No. Bernstein von-Mises Theorem is not applicable when the joint distribution does not have a differentiable second moment. And obviously joint i.i.d. Cauchy random variables does not even have a finite second moment. This condition requires a bounded energy assumption on the Riemannian manifold defined by the Rao-Fisher metric which is not satisfied by Cauchys.

2.Are there any (hopefully elementary) examples in which the Bayesian posterior is not asymptotically normal? In particular are there examples where $\pi,f$ are continuously differentiable with respect to $\theta$? $\pi(\theta)>0 $ for all $\theta$?

Yes. Indeed, we can choose an (noninformative) improper prior $\pi\propto C_0$ making the posterior also improper. For example $f\propto C_1$ is a trivial example. An improper posterior cannot be normal. For example, [Rubio&Steel] (14) provided an example where Jeffereys prior leading to an improper posterior which cannot be normal no matter how large the sample size is.

Reference

[Rubio&Steel]Rubio, Francisco J., and Mark FJ Steel. "Inference in two-piece location-scale models with Jeffreys priors." Bayesian Analysis 9.1 (2014): 1-22.

$\endgroup$
  • $\begingroup$ Thank you Henry.L, this is very useful, I will look up the reference. I am glad the question finally received some attention! $\endgroup$ – Joris Bierkens Mar 7 '17 at 14:05
  • $\begingroup$ Can you give a simple example with a proper prior? $\endgroup$ – Cagdas Ozgenc May 13 '17 at 6:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.