Checking my understanding of Bayes factor hypothesis testing

Question

I am new to Bayesian statistics. I am trying to understand my course notes on this topic. Here are the notes:

I will try to explain what these notes are saying and hopefully someone can correct my misunderstandings.

The opening two lines are relatively straightforward, because in Bayesian statistics we conceive of $\theta$ as a random variable, and we select whichever hypothesis has the higher probability, conditional on the data we've observed.

Regarding the Bayes factors, the Bayes factor against $H_0$ is

$$B = \frac{\frac{P(\theta \in \Theta_1 | y)}{P(\theta \in \Theta_0 | y)}}{\frac{P(\theta \in \Theta_1)}{P(\theta \in \Theta_0)}}.$$

(I found the $B_{10(y)}$ notation very confusing.) We choose $H_1$ if $B > 1$, or $\log_{10} B > 0$. Intuitively we are looking for the alternative hypothesis to gain on the null hypothesis when we update for the observed data.

Regarding the table of values given, I think I am supposed to reject $H_0$ for all these values. I found the table confusing because I guess my reflexes are more frequentist so when I read "evidence against $H_0$ is poor" I imagine something like a $p$-value of $0.09$, which is supposed to imply a level of evidence against $H_0$ that is non-zero but still too weak to reject $H_0$. However with this Bayes factor method, we more readily reject $H_0$ because we just require $H_1$ to be "better" (according to the definition of $B$) than $H_0$.

Am I making sense? I appreciate any help.

Answer 1

Rejecting hypotheses is mainly about making decisions under uncertainty. What counts as a reasonable threshold for rejection depends on what your goals are and your tolerance for potentially getting it wrong. A p-value of .09 might be too high to "reject" a null in some applications but acceptable in others, and you might very well consider a BF of 1.01 to be too close for your purposes.

With a BF of 1.01 (or p = .09), would you bet somebody lunch tomorrow against the null? Lunch for a whole week? Would you try to publish a paper based on that BF? What about strapping astronauts onto a million liters of liquid oxygen and blasting them into space? Etc etc.

So I think your instincts are roughly right here. BFs don't work quite the same way as p-values, but there's nothing magical about BFs such that nulls are automatically "rejectable" if there is any evidence against them at all. "How much" evidence often matters quite a lot.

---

There's another way to look at Bayes factors you might find interesting called the Savage-Dickey ratio.[1] SDR is the ratio of the posterior mass to prior mass on the null hypothesis:

$$BF_{01} = \frac{p(\theta = \theta_{null} | D)}{p(\theta = \theta_{null})}$$

(Written as BF in favor of the null). If the posterior assigns less mass to the null than the prior, then the evidence disfavored the null.

---

[1] See Wagenmakers 2010 (PDF) Appendix A for detail.