# When should I be worried about the Jeffreys-Lindley paradox in Bayesian model choice?

I am considering a large (but finite) space of models of varying complexity which I explore using RJMCMC. The prior on the parameter vector for each model is fairly informative.

1. In what cases (if any) should I be worried about the Jeffreys-Lindley paradox favoring simpler models when one of the more complex models would be more suitable?

2. Are there any simple examples which highlight the problems of the paradox in Bayesian model choice?

I have read a few articles, namely Xi'an's blog and Andrew Gelman's blog, but I still don't quite understand the problem.

• I think there are too many questions and they are too distinct to be effectively answered here. – jaradniemi Apr 21 '15 at 15:51
• Thanks for the feedback, @jaradniemi, I've removed the question "Should the RJMCMC procedure, which effectively returns posterior model probabilities, favor the same models as DIC would?" – Jeff Apr 21 '15 at 16:05

The Jeffreys-Lindley paradox is related to Bayesian model choice in that the marginal likelihood $$m(x)=\int \pi(\theta) f(x|\theta)\,\text{d}\theta$$ becomes meaningless when $\pi$ is a $\sigma$-finite measure (i.e., a measure with infinite mass) rather than a probability measure. The reason for this difficulty is that the infinite mass makes $\pi$ and $\mathfrak{c}\pi$ undistinguishable for any positive constant $\mathfrak{c}$. In particular, the Bayes factor cannot be used and should not be used when one model is endowed with a "flat" prior.
The original Jeffreys-Lindley paradox uses the normal distribution as an example. When comparing the models $$x\sim\mathcal{N}(0,1)$$ and $$x\sim\mathcal{N}(\theta,1)$$ the Bayes factor is $$\mathfrak{B}_{12}=\dfrac{\exp\{-n(\bar{x}_n)^2/2\}}{\int_{-\infty}^{+\infty}\exp\{-n(\bar{x}_n-\theta)^2/2\}\pi(\theta)\,\text{d}\theta}$$ It is well defined when $\pi$ is a proper prior but if you take a Normal prior $\mathcal{N}(0,\tau^2)$ on $\theta$ and let $\tau$ go to infinity, the denominator goes to zero for any value of $\bar{x}_n$ different from zero and any value of $n$. (Unless $\tau$ and $n$ are related, but this gets more complicated!) If instead you use directly $$\pi(\theta)=\mathfrak{c}$$where $\mathfrak{c}$ is a necessarily arbitrary constant, the Bayes factor $\mathfrak{B}_{12}$ will be $$\mathfrak{B}_{12}=\dfrac{\exp\{-n(\bar{x}_n)^2/2\}}{\mathfrak{c}\int_{-\infty}^{+\infty}\exp\{-n(\bar{x}_n-\theta)^2/2\}\,\text{d}\theta}=\dfrac{\exp\{-n(\bar{x}_n)^2/2\}}{\mathfrak{c}\sqrt{2\pi/n}}$$ hence directly dependent on $\mathfrak{c}$.