I have a question regarding model comparison using Bayes factors. In many cases, statisticians are interested on using a Bayesian approach with improper priors (for example some Jeffreys priors and reference priors).

My question is, in those cases where the posterior distribution of the model parameters is well-defined, is it valid to compare models using Bayes factors under the use of improper priors?

As a simple example consider comparing a Normal model vs. a Logistic model with Jeffreys priors.

  • $\begingroup$ An improper prior plays the role of a "noninformative prior". If you are in a "no prior belief" perspective then obviously you cannnot assign a prior probability to a model. However there are some papers by Berger & other authors about a notion of "intrinsic Bayes factors"; this sounds like Bayes factor with noninformative priors but I can't say more because I've never read these papers. There also probably exist other "objective Bayesian model selection" methods (typing these terms in Google yields several papers by Berger). $\endgroup$ – Stéphane Laurent Sep 6 '12 at 21:43
  • $\begingroup$ @StéphaneLaurent The interpretation of the prior on the parameters is different from that of the prior probability of the model. This can be seen from the general expression for the Bayes factor. You can also assign uniform priors to the models, improper prior to the parameters, and see what the data tells you a posteriori. $\endgroup$ – Jeffrey Sep 7 '12 at 10:39
  • 1
    $\begingroup$ I recommend reading Criteria for Bayesian model choice with application to variable selection (AoS, 2012), particularly Lemma 1. Basically, improper priors cannot be used for noncommon parameters. $\endgroup$ – user10525 Sep 19 '12 at 16:27

No. While improper priors can be okay for parameter estimation under certain circumstances (due to the Bernstein–von Mises theorem), they are a big no-no for model comparison, due to what is known as the marginalization paradox.

The problem, as the name would suggest, is that the marginal distribution of an improper distribution is not well-defined. Given a likelihood $p_1(x \mid \theta)$ and a prior $p_1(\theta)$: the Bayes factor requires computing the marginal likelihood:

$$p_1(x) = \int_\Theta p_1(x \mid \theta) p_1(\theta) d \theta .$$

If you think of an improper prior as being only known up to proportionality (e.g. $p_1(\theta) \propto 1$), then the problem is that $p_1(x)$ will be multiplied by an unknown constant. In a Bayes factor, you'll be computing the ratio of something with an unknown constant.

Some authors, notably E.T. Jaynes, try to get around this by defining improper priors as the limit of a sequence of proper priors: then the problem is that there may be two different limiting sequences that then give different answers.

  • 3
    $\begingroup$ Thank you for your answer. The issue about the proportionality constants can be avoided by using the same improper prior on common parameters, such as location and scale parameters, as mentioned in The Bayesian Choice pp. 349. If I properly understand, the marginalization paradox applies only to priors with a certain structure. $\endgroup$ – Jeffrey Sep 6 '12 at 15:53
  • $\begingroup$ The problem will be that unrealistic cases will dominate: if you have a uniform prior on your location parameter, you will be placing 100x the weight on the interval [100,200], as you would on [0,1] (which could seem ridiculous in some circumstances). $\endgroup$ – Simon Byrne Sep 6 '12 at 16:10
  • $\begingroup$ But the thing is that improper priors cannot be interpreted in probabilistic terms. There is no such weight given that the probabilistic interpretation of the prior is gone since it is improper. $\endgroup$ – Jeffrey Sep 6 '12 at 16:15
  • 2
    $\begingroup$ It's not probabilistic, but it is still a measure, so you can make relative comparisons (i.e. there is 100x the "mass" on the interval [100,200] as there is on [0,1]). $\endgroup$ – Simon Byrne Sep 6 '12 at 16:18
  • $\begingroup$ I think this analysis has to be done on the posterior rather than on the prior. For example, some matching priors are improper, such as the Independence Jeffreys for the Normal case $\pi(\mu,\sigma)\propto \sigma^{-1}$. You can apply that interpretation on this prior, but this prior produces posterior intervals with great frequentist properties. In this case, unrealistic cases do not dominate. (Thanks for the discussion, by the way) $\endgroup$ – Jeffrey Sep 6 '12 at 16:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.