I am considering a large (but finite) space of models of varying complexity which I explore using RJMCMC. The prior on the parameter vector for each model is fairly informative.

  1. In what cases (if any) should I be worried about the Jeffreys-Lindley paradox favoring simpler models when one of the more complex models would be more suitable?

  2. Are there any simple examples which highlight the problems of the paradox in Bayesian model choice?

I have read a few articles, namely Xi'an's blog and Andrew Gelman's blog, but I still don't quite understand the problem.

  • 1
    $\begingroup$ I think there are too many questions and they are too distinct to be effectively answered here. $\endgroup$ – jaradniemi Apr 21 '15 at 15:51
  • $\begingroup$ Thanks for the feedback, @jaradniemi, I've removed the question "Should the RJMCMC procedure, which effectively returns posterior model probabilities, favor the same models as DIC would?" $\endgroup$ – Jeff Apr 21 '15 at 16:05

Sorry for being unclear on my blog!

Note: I provided some background on Bayesian model choice and the Jeffreys-Lindley paradox in this other answer on Cross validated.

The Jeffreys-Lindley paradox is related to Bayesian model choice in that the marginal likelihood $$m(x)=\int \pi(\theta) f(x|\theta)\,\text{d}\theta$$ becomes meaningless when $\pi$ is a $\sigma$-finite measure (i.e., a measure with infinite mass) rather than a probability measure. The reason for this difficulty is that the infinite mass makes $\pi$ and $\mathfrak{c}\pi$ undistinguishable for any positive constant $\mathfrak{c}$. In particular, the Bayes factor cannot be used and should not be used when one model is endowed with a "flat" prior.

The original Jeffreys-Lindley paradox uses the normal distribution as an example. When comparing the models $$x\sim\mathcal{N}(0,1)$$ and $$x\sim\mathcal{N}(\theta,1)$$ the Bayes factor is $$\mathfrak{B}_{12}=\dfrac{\exp\{-n(\bar{x}_n)^2/2\}}{\int_{-\infty}^{+\infty}\exp\{-n(\bar{x}_n-\theta)^2/2\}\pi(\theta)\,\text{d}\theta}$$ It is well defined when $\pi$ is a proper prior but if you take a Normal prior $\mathcal{N}(0,\tau^2)$ on $\theta$ and let $\tau$ go to infinity, the denominator goes to zero for any value of $\bar{x}_n$ different from zero and any value of $n$. (Unless $\tau$ and $n$ are related, but this gets more complicated!) If instead you use directly $$\pi(\theta)=\mathfrak{c}$$where $\mathfrak{c}$ is a necessarily arbitrary constant, the Bayes factor $\mathfrak{B}_{12}$ will be $$\mathfrak{B}_{12}=\dfrac{\exp\{-n(\bar{x}_n)^2/2\}}{\mathfrak{c}\int_{-\infty}^{+\infty}\exp\{-n(\bar{x}_n-\theta)^2/2\}\,\text{d}\theta}=\dfrac{\exp\{-n(\bar{x}_n)^2/2\}}{\mathfrak{c}\sqrt{2\pi/n}}$$ hence directly dependent on $\mathfrak{c}$.

Now, if your priors are informative (and hence proper), there is no reason for the Jeffreys-Lindley paradox to occur. With a sufficient number of observations, the Bayes factor will consistently selected the model that generated the data. (Or more precisely the model within the collection of models considered for model choice that is closest to the "true" model that generated the data.)

  • 2
    $\begingroup$ Many thanks for your very detailed answer, Xi'an! Your blog is very clear (I have learned a lot from it) I was just a little slow in understanding this particular problem! $\endgroup$ – Jeff Nov 11 '15 at 16:32
  • $\begingroup$ Actually, my blog operates with highly variable assumptions on background and prerequisite, so it is certainly unclear at times and to many readers! $\endgroup$ – Xi'an Nov 11 '15 at 16:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.