Bayes factors denote how well a certain model is supported. Say that I am running a controlled experiment and I have two models: the null model and the alternative model.

If I have a high Bayes factor, could I argue that the treatment is effective and propose making the change?

  • 1
    $\begingroup$ Could you go into more details? What exactly is your decision-making scenario? What two models do you have? What is unclear about using Bayes factors for you? $\endgroup$ – Tim Mar 27 '16 at 18:43
  • $\begingroup$ Whether or not to manufacture a new medication is the decision scenario. We want to determine whether this medication was actually effective. $\endgroup$ – user46925 Mar 27 '16 at 18:44
  • 3
    $\begingroup$ Note: Bayes Factors can be alarmingly sensitive to the details of uninformative priors (and undefined for those that are improper). However, the sketched drug testing scenario also invites a simpler inference problem set within a single model that has a parameter representing the effect of drug treatment. That way you'll get a credible interval for the effect size as a bonus. $\endgroup$ – conjugateprior Mar 27 '16 at 20:31
  • 2
    $\begingroup$ I don't understand why somebody voted to close this as unclear. I think it's perfectly clear and the answer is basically Yes, but of course with some caveats as e.g. pointed out by @conjugateprior (+1). However, the first sentence of your question ("Bayes factors denote how well a certain model is supported") is wrong: Bayes factors are for comparing two models. $\endgroup$ – amoeba says Reinstate Monica Mar 27 '16 at 20:48
  • 2
    $\begingroup$ No, there are no 'Bayesian estimators' or even, strictly speaking 'Bayesian estimation' (though there are estimators that can have a Bayesian motivation). There is, on the other hand, Bayesian inference. But what you get from that is not an estimator, or even an estimate, but a joint distribution for all unknown quantities that are mentioned in a model (a.k.a the posterior) conditioned on the data. $\endgroup$ – conjugateprior Mar 28 '16 at 2:01

This is an excellent and deep question.

While traditional textbooks (like mine) tend to promote Bayes factors as equivalent to posterior probabilities of the null and alternative hypotheses or of two models under comparison, which is formally correct as detailed in the following extract from my Bayesian Choice, I now tend to think that the Bayes factor per se should not be used for decision-making but rather as a measure of relative evidence of one model versus the other. For instance, using $\mathfrak{B}^\pi_{01}(x)=1$ as the dividing line between null and alternative (or between model a and model b) does not strike me as a natural choice. Furthermore, I do not think the 0-1 loss advocated by Neyman and Pearson and later adopted by almost everyone is making much sense and brings any support to the decisional interpretation of the Bayes factor.

My current perspective on the Bayes factor is more in a prior or posterior predictive mode where the behaviour of $\mathfrak{B}^\pi_{01}(x)$ is assessed under both models, in order to calibrate the observed value $\mathfrak{B}^\pi_{01}(x)$ against both prior or posterior distributions of $\mathfrak{B}^\pi_{01}(x)$. This gets us away from the decisional perspective.

[From The Bayesian Choice, 2007, Section 5.2.2, page 227]

From a decision-theoretic point of view the Bayes factor is only a one-to-one transform of the posterior probability, but this notion came out to be considered on its own ground in Bayesian testing.

The Bayes factor is the ratio of the posterior probabilities of the null and the alternative hypothesis over the ratio of the prior probabilities of the null and the alternative hypothesis, i.e., $$ \mathfrak{B}^\pi_{01}(x) = {\mathbb{P}(\theta \in \Theta_ 0\mid x) \over \mathbb{P}(\theta \in \Theta_1\mid x)} \bigg/ {\pi(\theta \in \Theta_ 0) \over \pi(\theta \in \Theta_ 1)}. $$

This ratio evaluates the modification of the odds of $\Theta_0$ against $\Theta_1$ due to the observation(s) and can naturally be compared to $1$, although an exact comparison scale can only be based upon a loss function.

The Bayes factor is, from a Bayesian decision-theoretic point of view, completely equivalent to the posterior probability of the null hypothesis as $H_0$ is accepted when $$ B^\pi_{01} (x) \ge {a_1\over a_0} \big/ {\rho_0 \over \rho_1} = {a_1\rho_1 \over a_0\rho_0}, $$ where $$ \begin{align*} \rho_0 &= \pi(\theta\in\Theta_0) \quad \hbox{ and } \nonumber\\ \rho_1 &= \pi(\theta\in\Theta_1)\\ &=1-\rho_0. \end{align*} $$

and where $a_0$ and $a_1$ are the penalties for wrongly selecting the alternative and null hypotheses or the models $\mathfrak{M}_0$ and $\mathfrak{M}_1$. respectively, in Neyman-Pearson formulation: $$ \mathfrak{L}(\theta, \varphi) = \begin{cases} 0 &\text{if $\varphi=\mathbb{I}_{\Theta_0}(\theta)$,} \cr a_0 &\text{if $\theta\in\Theta_0$ and $\varphi=0$,} \cr a_1 &\text{if $\theta\not\in\Theta_0$ and $\varphi=1$,}\cr\end{cases} $$

  • 1
    $\begingroup$ (+1) Maybe you should add that $a_0$ and $a_1$ are the penalties associated to errors when the null is true or false. $\endgroup$ – peuhp Mar 29 '16 at 14:01
  • 1
    $\begingroup$ Amazing answer Xi'an. Thanks for the response, I really appreciate it. $\endgroup$ – user46925 Mar 29 '16 at 14:08
  • 1
    $\begingroup$ +1. I formatted the quote from your textbook as a quote -- apologies if you did not want to have it like that for some reason (feel free to revert my edit). I also removed the words "Panayiota Touloupou" that somehow got into the definition during the last revision. $\endgroup$ – amoeba says Reinstate Monica Mar 29 '16 at 15:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy