Bayes Factor vs Likelihood Ratio Calculation

Question

Given data $D$ and models $M_i$ for $i=1,2,\ldots,n$ with parameter $\theta$, in a Bayesian setting why do we integrate over the parameter space $$ \mathbb{P}(D|M_i) = \int p(D|\theta;M_i) \text{ } p(\theta|M_i) \text{ } d\theta $$ when computing the Bayes Factor whereas in a frequentist setting we maximize over the parameter space when computing the Likelihood Ratio?

Answer 1

I think the original poster (OP) is asking why Bayesian model comparison (using Bayes factors) marginalizes across the entire parameter space, but likelihood ratio tests consider only the isolated points in the two models' parameter spaces that maximize the fit.

The answer is: The two approaches measure model fit differently, and that difference comes down to the role of the prior distribution. In a likelihood ratio test, only the best parameter point is considered to be "in" the model. In a Bayesian approach, the prior distribution is inherently part of the model.

In a likelihood ratio test, there is no prior distribution on parameters, and the (maximum) likelihood of the data for model $m$ is the probability of the data at whatever point in parameter space maximizes the the probability. All points in parameter space are equally available candidates for fitting the data, but only the best one is actually considered to be "in" the best fit.

In a Bayesian approach, on the other hand, the prior distribution is inherently part of the model. You can't specify a model without specifying the probability distribution across the parameter space. A model with a prior that loads a lot of probability mass over parameter values that nicely fit the data is a better model than a model with a prior that loads a lot of probability mass over parameter values that don't fit the data.

Answer 2

It's a difference of philosophy. Firstly, the "generalized" likelihood ratio test should not be confused with the standard likelihood ratio test. the standard test has strong optimality properties, which are only really possible when testing simple (non-composite) hypothesis. The generalized LR test is based on a asymptotic result, known as Wilks' theorem, which shows that the log (generalized) likelihood ratio converges to a Chi-square distribution as the sample size goes to infinity, under the null hypothesis. Assuming this distribution, the null hypothesis is rejected if the probability of seeing a result as extreme or more extreme than the measured result is unlikely, say with a threshold of 5% chance. Most frequentist tests have this form, where the test statistic's distribution, either asymptotically or non-asymptotic, follows a particular distribution if the null hypothesis is true.

In contrast, the bayes factor test makes stronger distributional assumptions up front (i.e. priors on parameters), and this allows it to make strong non-asymptotic statements about the evidence for one hypothesis other another. It relies crucially on the interpretation of probabilities as beliefs, and so is incompatible with frequentist notions of probabilities as rates of long term repeated events.

Both approaches have positive and negative attributes. Neither approach is clearly the best, since there is no clear criterion to judge which test is better than another. Likelihood ratio tests are far more popular in practice though.

Answer 3

I don't think OP is talking about the standard likelihood ratio test that one would use to compute a p-value. I believe s/he is talking about the generalized test. One could argue that the generalized test has more in common with Bayesian statistics than it does with frequentist, as one who uses the generalized test might be called a 'likelihoodist'.

Although the typical use of the generalized test would be to compare a model based on the observations to a null model, and would give answers related to p-values (albeit much more intuitive ones), the approach also allows one to specify two predictive models a priori, neither of which would depend on the observations for their point estimates (though the likelihood ratio would still normally be computed assuming the models share the variance found in the observations).

In this sense, likelihood ratios are model comparison approaches just as Bayesian analyses are, and as such have the same desirable property of being statistically symmetrical inasmuch as the evidence can be used to support either model. In comparison, p-values using NHST test only one model (the null) and infer the evidence for the alternative model based on the evidence against the null. Further, evidence for the null itself is logically unobtainable as one can only 'fail to reject' it, never 'accept' it. It is a very odd way of testing models.

As far as the differences between parametrization of likelihood ratios and Baeysian models are concerned, the main difference is that only the Bayesian model assumes a prior distribution, and use the data to update one's prior beliefs. The likelihood ratio instead makes no such assumption but simply tests the two models. John Kruschke above explained how the parameter space is used by each model in a clear way. The only thing I would add to that is that the LR does not necessarily need to use the observed data to set the parameters of the alternative model, but could use any pre-specified value, for example, the size of an effect predicted by a theoretical model, which would not necessarily match the size observed in the data.