12
$\begingroup$

I am trying to understand Bayes Factor (BF). I believe they are like likelihood ratio of 2 hypotheses. So if BF is 5, it means H1 is 5 times more likely than H0. And value of 3-10 indicates moderate evidence, while >10 indicates strong evidence.

However, for P-value, traditionally 0.05 is taken as cut-off. At this P value, H1/H0 likelihood ratio should be about 95/5 or 19.

So why a cut-off of >3 is taken for BF while a cut-off of >19 is taken for P values? These values are not anywhere close either.

$\endgroup$
  • 2
    $\begingroup$ I am uncomfortable with saying "if BF is $5$, it means $H_1$ is $5$ times more likely than $H_0$". The Bayes factor may be a marginal likelihood ratio, but it is not a probability ratio or odds ratio, and needs to be combined with a prior to be useful $\endgroup$ – Henry Apr 25 at 10:17
  • $\begingroup$ If we do not have any particular prior information, then what can we say about meaning of BF? $\endgroup$ – rnso Apr 25 at 11:26
  • $\begingroup$ Certainly, one has "some" prior information even if saying that there is no any particular prior info. Namely, in that case it's reasonable to assign equal probabilities to each hypothesis according to the principle of indifference. That is a simple example of a so called non-informative prior (admittedly a misnomer). $\endgroup$ – dnqxt Apr 25 at 18:20
  • $\begingroup$ In this case will BF of 5 indicate one hypothesis to be 5x more likely? $\endgroup$ – rnso Apr 25 at 18:36
  • $\begingroup$ Yes, but this problem is much more complicated than it might seem and goes into the area of model selection in statistics. You've been warned :)) $\endgroup$ – dnqxt Apr 25 at 18:43
10
$\begingroup$

A few things:

The BF gives you evidence in favor of a hypothesis, while a frequentist hypothesis test gives you evidence against a (null) hypothesis. So it's kind of "apples to oranges."

These two procedures, despite the difference in interpretations, may lead to different decisions. For example, a BF might reject while a frequentist hypothesis test doesn't, or vice versa. This problem is often referred to as the Jeffreys-Lindley's paradox. There have been many posts on this site about this; see e.g. here, and here.

"At this P value, H1/H0 likelihood should be 95/5 or 19." No, this isn't true because, roughly $p(y \mid H_1) \neq 1- p(y \mid H_0)$. Computing a p-value and performing a frequentist test, at a minimum, does not require you to have any idea about $p(y \mid H_1)$. Also, p-values are often integrals/sums of densities/pmfs, while a BF doesn't integrate over the data sample space.

$\endgroup$
  • 2
    $\begingroup$ Taylor is saying the threshold for evidence against one hypothesis ($\text{H}_0$) can't be directly compared to the threshold of evidence for another hypothesis ($\text{H}_1$), also not approximately. When you stop believing in a null-effect need not relate to when you start believing in an alternative. This is exactly why the $p$-value shouldn't be interpreted as $1 - (\text{belief in H}_1)$ $\endgroup$ – Frans Rodenburg Apr 25 at 7:27
  • 1
    $\begingroup$ Maybe this can be clarifying: en.wikipedia.org/wiki/Misunderstandings_of_p-values The frequentist $p$-value is not a measure of evidence for anything. $\endgroup$ – Frans Rodenburg Apr 25 at 7:33
  • 2
    $\begingroup$ Sorry, last comment: The reason you can't see it as evidence in favor of $\text{H}_1$ is that it is the chance of observing this large an effect size if $\text{H}_0$ were true. If $\text{H}_0$ is indeed true, the $p$-value should be uniformly random, so its value has no meaning on the probability of $\text{H}_1$. This subtlety in interpretation is by the way one of the reasons $p$-values see so much misuse. $\endgroup$ – Frans Rodenburg Apr 25 at 7:44
  • 1
    $\begingroup$ @benxyzzy: the distribution of a $p$-value is only uniform under the null hypothesis, not under the alternative where it is heavily skewed towards zero. $\endgroup$ – Xi'an Apr 25 at 12:23
  • 1
    $\begingroup$ @benxyzzy To add to others: The point of using a $p$-value is that under null hypothesis it is uniformly random, so if you get a very small $p$-value, it hints that maybe it wasn't uniformly random so maybe the null hypothesis wasn't true. $\endgroup$ – JiK Apr 25 at 13:50
8
$\begingroup$

The Bayes factor $B_{01}$ can be turned into a probability under equal weights as $$P_{01}=\frac{1}{1+\frac{1}{\large B_{01}}}$$but this does not make them comparable with a $p$-value since

  1. $P_{01}$ is a probability in the parameter space, not in the sampling space
  2. its value and range depend on the choice of the prior measure, they are thus relative rather than absolute (and Taylor's mention of the Lindley-Jeffreys paradox is appropriate at this stage)
  3. both $B_{01}$ and $P_{01}$ contain a penalty for complexity (Occam's razor) by integrating out over the parameter space

If you wish to consider a Bayesian equivalent to the $p$-value, the posterior predictive $p$-value (Meng, 1994) should be investigated $$Q_{01}=\mathbb P(B_{01}(X)\le B_{01}(x^\text{obs}))$$ where $x^\text{obs}$ denotes the observation and $X$ is distributed from the posterior predictive $$X\sim \int_\Theta f(x|\theta) \pi(\theta|x^\text{obs})\,\text{d}\theta$$ but this does not imply that the same "default" criteria for rejection and significance should apply to this object.

$\endgroup$
  • $\begingroup$ Using your formula, P for BF of 3 and 10 come out to be 0.75 and 0.91, respectively. Why should we accept these as moderate evidence since for P value we keep cut-off of 0.95 ? $\endgroup$ – rnso Apr 26 at 7:44
  • $\begingroup$ Why is $0.95$ relevant in this framework? or at all? Deciding when large is large enough depends on your utility function. $\endgroup$ – Xi'an Apr 26 at 9:56
  • $\begingroup$ The formula looks simpler as P = B/(B+1) $\endgroup$ – rnso Apr 26 at 10:12
2
$\begingroup$

Some of your confusion might stem from taking the number 95/5 directly from the fact that the p value is 0.05 - is this what you are doing? I do not believe this is correct. The p value for a t-test, for example, reflects the chance of getting the observed difference between means or a more extreme difference if the null hypothesis is in fact true. If you get a p value of 0.02, you say 'ah, there is only a 2% chance of getting a difference like this, or a greater difference, if the null is true. That seems very improbable, so I propose that the null is not true!'. These numbers are just not the same thing that goes into the Bayes factor, which is the ratio of the posterior probabilities given to each competing hypothesis. These posterior probabilities are not computed in the same way as the p-value, and so thinking of 95/5 as being like posterior probabilities that would give a BF of 19 is not correct.

As a side note, I would suggest strongly guarding against thinking of different BF values as meaning particular things. These assignments are completely arbitrary, just like the .05 significance level. Problems such as p-hacking will occur just as readily with Bayes Factors if people start to believe that only particular numbers warrant consideration. Try to understand them for what they are, which are something like relative probabilities, and use your own sense to determine whether you find a BF number convincing evidence or not.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.