The posterior odds is the ratio of the Bayes factor $\times$ prior odds of the hypotheses.

$\frac{p(H_0 | D)}{p(H_1 | D)}$ = $\frac{p(D | H_0)}{p(D | H_1)} \frac{p(H_0)}{p(H_1)}$

It is considered a difficult number to calculate because $p(D | H_i)$ , we would need to calculate $\int p(D | H_i, \theta_i) \pi(\theta_i | H_i) d\theta_i$, the marginal likelihood of hypothesis $i$. Here, $\pi(\theta_i | H_i)$ is the prior on the parameters for hypothesis $i$.

Question: Why is the hypothesis the entire model class? If I were to say I think the log of heights of a population are Gaussian distributed as opposed to something longer-tailed, like Cauchy, why do I need to consider ALL Gaussians? Why can't I just compare the Gaussian at maximum likelihood? Why can't $H_i$ stand for the model class at its optimal parameters?

  • $\begingroup$ These questions amount to "why using a Bayesian approach". The debate has been going on for centuries. $\endgroup$ – Xi'an Oct 18 '18 at 10:56
  • $\begingroup$ Even if we specify the hypothesis to just the optimal value of the parameters, we'd still include a prior on the hypothesis, so it would still retain its Bayesian-ness, wouldn't it? Or would that just pretty much be a likelihood ratio? $\endgroup$ – ken Oct 18 '18 at 11:01
  • $\begingroup$ The hypothesis cannot be reduced to the estimated value of the parameter, since it could not be defined prior to seeing the data. $\endgroup$ – Xi'an Oct 18 '18 at 16:47

The Bayes factor $B^\pi_{01} (x)$ is, from a decision-theoretic point of view, under $a_0-a_1$ loss, that is, penalising the (wrong) decision of choosing $H_0$ when $H_1$ is true by $a_0$, the (wrong) decision of choosing $H_1$ when $H_0$ is true by $a_1$, and the (right) decision of choosing $H_i$ when $H_i$ is true by $0$, completely equivalent to the posterior probability of the null hypothesis as $H_0$ is accepted when and only when \begin{equation} B^\pi_{01} (x) \ge {a_1\over a_0} \Big/ {\rho_0 \over \rho_1} = {a_1\rho_1 \over a_0\rho_0}, \end{equation} where the prior weights are defined as \begin{eqnarray} \rho_0 &=& \pi(\theta\in\Theta_0) \quad \hbox{ and } \nonumber\\ \rho_1 &=& \pi(\theta\in\Theta_1)=1-\rho_0. \end{eqnarray} Thus this is a Bayesian optimal decision, in the above decision theoretic sense, which should suffice by itself as a justification.

In the case when $H_0$ and $H_1$ are made of a single parameter value each, the Bayes factor of course reduces to the likelihood ratio. Otherwise, it weights the likelihood according to the prior for each hypothesis, resulting in $$B^\pi_{01} (x)=\int_{\Theta_0} \pi_0 (\theta) f(x|\theta) d\theta \Bigg/ \int_{\Theta_1} \pi_1 (\theta) f(x|\theta) d\theta$$ as an averaged likelihood ratio (or rather a ratio of averaged likelihoods). This is coherent from a Bayesian view that all parameters are integrated out and weighted by the prior, much more that plugging in an estimate for each hypothesis since this is a random variable and its variability impacts the denominator differently from the numerator. It is with good reasons that the Bayes factor is seen as a modern version of Ockham's razor. More advanced arguments show that integrating out the parameter penalises the dimension of the hypothesis (or model) as in the BIC or Swartz' criterion, while taking the ratio of two likelihoods always favours the most complex of the two hypotheses (or models).

  • $\begingroup$ What is an $a_0 - a_1$ loss? and what is meant by "Bayesian optimal solution" $\endgroup$ – ken Oct 18 '18 at 13:35
  • $\begingroup$ Also, is "posterior odds" used as a criteria for selecting the hypothesis, or the "Bayes factor"? What I had a problem with, I guess if we are looking at posterior odds, is the flat prior used for $\Theta_i$s, because it would seem to equally weight into the hypothesis, parameter values which might be completely nonsensical for the data. $\endgroup$ – ken Oct 18 '18 at 13:38
  • $\begingroup$ H, ok so I've digested this a bit more. In the BF, there is a weighting of the likelihood by the parameter prior - so here you can account for nonsensical parameters. And then the prior ratio of the hypothesis, is the ratio of the model class. I guess this makes sense. Now just the $a_0 - a_1$ loss and BOS I'm not really understanding yet. $\endgroup$ – ken Oct 18 '18 at 14:26
  • $\begingroup$ I added the definition of the loss function, but cannot start a crash course on Bayesian decision theory here when I can refer you to my book. $\endgroup$ – Xi'an Oct 18 '18 at 16:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.