Checking my understanding of Bayes factor hypothesis testing

I am new to Bayesian statistics. I am trying to understand my course notes on this topic. Here are the notes:

I will try to explain what these notes are saying and hopefully someone can correct my misunderstandings.

The opening two lines are relatively straightforward, because in Bayesian statistics we conceive of $$\theta$$ as a random variable, and we select whichever hypothesis has the higher probability, conditional on the data we've observed.

Regarding the Bayes factors, the Bayes factor against $$H_0$$ is

$$B = \frac{\frac{P(\theta \in \Theta_1 | y)}{P(\theta \in \Theta_0 | y)}}{\frac{P(\theta \in \Theta_1)}{P(\theta \in \Theta_0)}}.$$

(I found the $$B_{10(y)}$$ notation very confusing.) We choose $$H_1$$ if $$B > 1$$, or $$\log_{10} B > 0$$. Intuitively we are looking for the alternative hypothesis to gain on the null hypothesis when we update for the observed data.

Regarding the table of values given, I think I am supposed to reject $$H_0$$ for all these values. I found the table confusing because I guess my reflexes are more frequentist so when I read "evidence against $$H_0$$ is poor" I imagine something like a $$p$$-value of $$0.09$$, which is supposed to imply a level of evidence against $$H_0$$ that is non-zero but still too weak to reject $$H_0$$. However with this Bayes factor method, we more readily reject $$H_0$$ because we just require $$H_1$$ to be "better" (according to the definition of $$B$$) than $$H_0$$.

Am I making sense? I appreciate any help.

Rejecting hypotheses is mainly about making decisions under uncertainty. What counts as a reasonable threshold for rejection depends on what your goals are and your tolerance for potentially getting it wrong. A p-value of .09 might be too high to "reject" a null in some applications but acceptable in others, and you might very well consider a BF of 1.01 to be too close for your purposes.

With a BF of 1.01 (or p = .09), would you bet somebody lunch tomorrow against the null? Lunch for a whole week? Would you try to publish a paper based on that BF? What about strapping astronauts onto a million liters of liquid oxygen and blasting them into space? Etc etc.

So I think your instincts are roughly right here. BFs don't work quite the same way as p-values, but there's nothing magical about BFs such that nulls are automatically "rejectable" if there is any evidence against them at all. "How much" evidence often matters quite a lot.

There's another way to look at Bayes factors you might find interesting called the Savage-Dickey ratio.[1] SDR is the ratio of the posterior mass to prior mass on the null hypothesis:

$$BF_{01} = \frac{p(\theta = \theta_{null} | D)}{p(\theta = \theta_{null})}$$

(Written as BF in favor of the null). If the posterior assigns less mass to the null than the prior, then the evidence disfavored the null.

[1] See Wagenmakers 2010 (PDF) Appendix A for detail.

• Thanks for your response. After reading this, I would say the main issue with what's written in my course notes is the oversimplification that "we choose $H_1$ if $B_{10(y)} > 1$". As you argue, it depends on the context, and I agree with this. The reference to the Savage-Dickey ratio seems to highlight to me (as a relative neophyte in statistics) that there is more than one way to think about what constitutes (strong) evidence. Thanks again. Dec 6, 2020 at 6:30