How are Bayes factors actually Bayesian?

Question

I have been doing some linear model analyses involving Bayes factors lately and I have two probably very basic questions:

1) As far as I understand, a Bayes factor is simply a likelihood ratio, i.e.

p(data|M1)/p(data|M2).

But that's not really Bayesian inference, is it? Since the whole point of Bayesian inference is to convert p(data|model) to p(model|data)?

Sure, people argue that given equal prior probabilities of both models, the above equation is equivalent to p(M1|data)/p(M2|data), but still it seems to me like the Bayes factor approach is missing the whole point of Bayesian inference. Especially since the really cool thing about Bayesian modeling is that I can have both prior and posterior distributions for each model coefficient, I feel like Bayes factor model comparison is falling short of the power of Bayesian models?

2) How is it possible, in the first place, that Bayes factors, being based on (unpenalized) likelihood, can favor the null model? Shouldn't the likelihood always increase with more complex models, i.e. with the non-null model?

Hope some of you can shed a little light on my mind.

Answer 1

While I have written rather extensively on the limitations of the Bayes factor as a mean to conduct model comparison, the first point is that indeed it is not the Bayesian answer to the testing question Which of model $M_1$ or model $M_2$ is the true model? when evaluated with the Neyman-Pearson loss function. The proper Bayesian answer is the model with the largest posterior probability. When using other loss functions, the proper Bayesian answer is the posterior probability itself. Historically, the Bayes factor has been defended by Jeffreys (circa 1939) as providing a more objective answer than the posterior probability, as it evacuates the impact of the prior weights of the models $M_1$ and $M_2$, which can be deemed as failing to take advantage of the Bayesian formalism, but truly reflects the immense uncertainty in setting such weights.

On the second point, it is well-documented that Bayes factors are providing a natural penalisation for complexity by integrating out over a larger space, as opposed to likelihood ratios that exploit the best fits under both models. The Bayes factor is often seen as a quasi-automatic Occam's razor for this reason, which is also why the Bayesian information criterion (BIC) $$\mathrm{BIC} = {\log(n)k - 2\log(L({\hat \theta}))}$$ is used by non-Bayesians as well.

Answer 2

The issue is that while a Bayes factor, assuming equal prior probability of the competing hypotheses, is a likelihood ratio, it's not the likelihood ratio you think it is. Here, the likelihood of $M_1$ is

$$ p(data|M_1) = \int p(data|M_1, \theta_1) p(\theta_1|M_1) d\theta_1. $$

The trick you were missing that directly answers both parts 1) and 2) is that what we mean in this context when we say the likelihood of the data given model 1 incorporates the prior on the parameters -- the only (potential) prior information we (often) throw out is that we are just assuming (typically) that the hypotheses are a priori equally likely.

In other words, you were conflating $p(data|M_1)$ with $p(data|M_1, \theta_1)$.